[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-evaluation":3,"mdc--i5jai2-key":34,"related-org-nvidia-evaluation":7338,"related-repo-nvidia-evaluation":7498},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"evaluation","evaluate LLM accuracy with NeMo","Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on \"evaluate model\", \"benchmark accuracy\", \"run MMLU\", \"evaluate quantized model\", \"run nel\". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying\u002Fserving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Benchmarking","benchmarking","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},3227,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer","2026-08-31T09:19:35.777526","Apache-2.0",498,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Ftree\u002FHEAD\u002Fplugins\u002Fmodelopt\u002Fskills\u002Fevaluation","---\nname: evaluation\ndescription: Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on \"evaluate model\", \"benchmark accuracy\", \"run MMLU\", \"evaluate quantized model\", \"run nel\". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying\u002Fserving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).\nlicense: Apache-2.0\n# Based on nel-assistant skill from NeMo Evaluator Launcher (commit f1fa073).\n# https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Ftree\u002Ff1fa073\u002Fpackages\u002Fnemo-evaluator-launcher\u002F.claude\u002Fskills\u002Fnel-assistant\n---\n\n## NeMo Evaluator Launcher Assistant\n\nGuide the user through creating NEL YAML configs, running evaluations, and monitoring progress.\n\n### Workspace integration\n\nIf `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md` and reuse existing workspaces (this skill is usually the final stage of PTQ → Deploy → Eval; carry any deployment-time patches into `deployment.command`).\n\n### Workflow\n\n```text\n- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)\n- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)\n- [ ] Step 2: Build base config (5-question flow OR shortcut)\n- [ ] Step 3: Configure deployment (model path, params, cross-check)\n- [ ] Step 4: Fill remaining ??? values\n- [ ] Step 5: Confirm tasks (iterative)\n- [ ] Step 6: Multi-node (if needed)\n- [ ] Step 7: Interceptors (if needed)\n- [ ] Step 7.5: Container auth (SLURM private images)\n- [ ] Step 8: Dry-run → canary → full run\n- [ ] Step 9: Verify completed run\n```\n\n---\n\n### nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST\n\nA few **agentic** AA benchmarks do **not** run on the currently validated\n`nemo-evaluator-launcher` 0.2.6 path (Steps 1–9 don't apply). They run on **nel-next**\n(`nemo-evaluator[harbor]` 0.4.x) — a separate package, CLI (`nel eval run`), `-O`\noverrides, and `services`\u002F`benchmarks`\u002F`cluster`\u002F`output` schema. If the user asks\nfor one, do **not** add it to a 0.2.6 `evaluation.tasks` list — instead:\n\n1. Read **`references\u002Fnel-next.md`** (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe `recipes\u002Ftasks\u002Faa_next\u002F{terminal_bench_2_1,swebench_verified}.md`; start from `recipes\u002Fexamples\u002Fexample_eval_next.yaml`.\n2. Isolated nel-next venv: `\"$SKILL_DIR\u002Fscripts\u002Fnel-next.sh\" --setup-only` (keeps 0.2.6 `nel` untouched).\n3. Run **`modelopttools:eval-config`** (Step 3b) to write the AWS-sandbox creds + harbor infra rows (`${NEL_NEXT_EVAL_IMAGE}`, `${HARBOR_*_ECR_REPOSITORY}`) into `.env`; always include the `output.export_config.mlflow` block.\n4. Dry-run → canary → full (`nel-next.sh eval run`), then **push to MLflow** — SLURM doesn't auto-export, so run `nel-next.sh mlflow-push -r \u003Crun_id> -c \u003Ccfg>` after (config-driven; see `references\u002Fnel-next.md`).\n\nSteps 1–9 below are currently validated with 0.2.6 — use them for everything else.\n\n---\n\n### GDPVal (NeMo Gym \"Stirrup\" agent) path — branch here too\n\nGDPVal **does** run on the currently validated 0.2.6 `nel` launcher (as a\n`nemo_gym` task, not nel-next), so Steps 1–9 apply — but it is mechanically\nspecial and **standalone** (one gym eval per config; never mix it with `aa\u002F`\ntasks). If the user asks for GDPVal:\n\n1. Read **`references\u002Fgym-gdpval.md`** (Apptainer SIF sandbox, gym prepare\u002Freap\n   machinery, deploy sizing, rubric-vs-comparison scoring, MLflow deliverables trap,\n   failure modes) + **`recipes\u002Ftasks\u002Fgym\u002Fgdpval.md`**.\n2. Start from **`recipes\u002Fexamples\u002Fgym\u002Fexample_gdpval.yaml`** — a single\n   self-contained file.\n3. Prerequisite — the Apptainer SIF. **If your site provides one, use it**\n   (NVIDIA-internal: `modelopttools:eval-config` Step 3c); otherwise set\n   `GDPVAL_SIF_DIR` in `.env` and build with `\"$SKILL_DIR\u002Fscripts\u002Fgdpval-sif.sh\"`\n   (build-if-absent, no cross-cluster copy). Either way the mounted dir must contain\n   the file `GDPVAL_CONTAINER_PATH` names (template: `python-3.13.gdpval.sif`) — a\n   name mismatch passes NEL's `test -d` check and the agent then silently runs\n   unsandboxed. Verify with `gdpval-sif.sh --check`. `.env` needs `HF_TOKEN`, `INFERENCE_API_KEY`, `TAVILY_API_KEY`,\n   `INFERENCE_JUDGE_URL`, `GDPVAL_SIF_DIR`, and `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the\n   config has a `pre_cmd`). Thinking mode is mandatory (non-thinking loses ~86%).\n4. Run both dry-run and launch through `\"$SKILL_DIR\u002Fscripts\u002Fnel-gdpval.sh\"`; it\n   enforces the currently validated 0.2.6 launcher even if `nel` on PATH is stale\n   and avoids an unset `NEL_INVOCATION_ID` failure before client startup.\n   **`limit_samples` is inert on the gym path** (the gym runs all 220 tasks\n   regardless), so there is no cheap canary: watch the real run's first\n   ~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the\n   recipe's Canary section.\n\n---\n\n### MRCR (NeMo Gym `simple_agent`) path — branch here too\n\nA 0.2.6 `nemo_gym` task like GDPVal and equally **standalone**, but far simpler:\n`simple_agent`, **no SIF, no judge, no Tavily** — deterministic prefix-gated\ngrading, `HF_TOKEN` the only secret. **Not an AA benchmark** — never generate it\nfor an \"AA\" request. If the user asks for MRCR:\n\n1. Read **`recipes\u002Ftasks\u002Fgym\u002Fmrcr.md`**; start from\n   **`recipes\u002Fexamples\u002Fgym\u002Fexample_mrcr.yaml`** (1M variant, like the golden).\n2. **Pick the variant first** (`config_n3_1m` \u002F `config_n3_128k` \u002F `config`) — it\n   sets the context cap, dataset *and* metric prefix; the three are not\n   comparable; set it in **both** `data_prep_params` and `collect_rollout_params`.\n3. `.env`: `HF_TOKEN` (dataset + n3 tokenizer are gated) plus\n   `NEMO_EVALUATOR_TRUST_PRE_CMD=1` (the `pre_cmd` installs `tiktoken` +\n   `transformers`; prepare fails without it) and\n   `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1` (`nemo_gym` is not in the FDF map).\n4. **MRCR needs a git-backed Gym image.** The pin is newer than any image's baked\n   Gym and must apply, so the template's `container:` is `???` and the bootstrap\n   exits 1 on a non-git `\u002Fopt\u002FGym` (the public `eval-factory\u002Fnemo-gym:*` images).\n   NVIDIA-internal: `modelopttools:eval-config` Step 3d names a working image.\n5. Long-context deploy (`--max-model-len 1100000` +\n   `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`, `gpu_memory_utilization: 0.95`,\n   multi-instance fan-out); **never cap output tokens**; report the needle-count\n   strata alongside `pass@1\u002Faccuracy`.\n\n---\n\n### Step 1 — Prerequisites\n\n> **SSH:** NEL opens its own connection as a bare `ssh \u003Cuser>@\u003Chost>` with **no `-i`**, so a\n> non-default key filename makes every submit fail with `Permission denied (publickey,password)`\n> even though your own tooling works. Add an `IdentityFile` entry to `~\u002F.ssh\u002Fconfig` per cluster and\n> verify with `ssh -o BatchMode=yes \u003Cuser>@\u003Chost> true`.\n\n> **Config placement gotchas**, each of which costs one failed submission: `sbatch_comment` belongs\n> under `execution:`, not `cluster:` (otherwise inert, and the idle-GPU reaper kills the job); pyxis\n> needs `registry#path:tag` for non-DockerHub images or it prepends `docker.io` and 404s; NEL rejects\n> **file** mounts (\"Mount paths must be directories\") — put file overrides in `pre_cmd`.\n\nRun `nel --version`; if missing, instruct `pip install nemo-evaluator-launcher`. If user has an existing config, skip to Step 8 (optionally review for `???` and quantization flags first).\n\n**Set up `.env` now (not Step 8).** The working `.env` lives at the **workspace root** — the directory you run `nel` from — matching `modelopttools:eval-config`'s convention; do **not** create it under the skill dir. (NEL does not discover `.env` by path: it reads secrets from the shell env via the `host:` prefix after you `source`, so the location is purely *which file you source* before `nel run`. Keeping the single `.env` at the workspace root avoids a stale duplicate under the symlinked, shared `.agents\u002F` skill tree.) For judge-scored \u002F user-sim tasks (HLE, AA-LCR, Tau2), seed it from the template if absent — the template ships under the skill dir, the working `.env` does not: `[ -f .env ] || cp \"$SKILL_DIR\u002Frecipes\u002Fenv.example\" .env`. Then try `modelopttools:eval-config` (if available) to fill the judge `model_id`\u002F`url` rows (user adds the secret key). Needed before Step 5, which substitutes those values into task `\u003CVAR>` placeholders.\n\n**Secret safety — never open `.env` with Read\u002FWrite\u002FEdit.** The harness mirrors later edits of any agent-opened file into the transcript, so touching `.env` leaks the keys the user adds afterward. Use shell only (`cp` to create, `source` to load — neither echoes); edit `env.example`, never `.env`; leave value entry to the user \u002F `modelopttools:eval-config`.\n\n**Task recipes** (always read before editing the relevant task in the config):\n\n- AA Index v2 suite (default for quantized-checkpoint validation, see `references\u002Fquantization-benchmarks.md`): `recipes\u002Ftasks\u002Faa\u002F{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom,omniscience}.md`\n- Optional: `recipes\u002Ftasks\u002Fmmlu_pro.md`, `recipes\u002Ftasks\u002Faime_2025.md`, `recipes\u002Ftasks\u002Flivecodebench.md`\n- **nel-next only** (different evaluator — see the nel-next section below, NOT the 0.2.6 steps): shared reference `references\u002Fnel-next.md` + per-benchmark recipes `recipes\u002Ftasks\u002Faa_next\u002F{terminal_bench_2_1,swebench_verified}.md` (agentic). The `aa_next\u002F` dir holds tasks that require `nemo-evaluator[harbor]` 0.4.x (the package; `nemo-evaluator-next` is the eval *image* repo); `aa\u002F` is the 0.2.6 suite.\n- **NeMo Gym tasks** — `recipes\u002Ftasks\u002Fgym\u002F*.md`, with self-contained examples at `recipes\u002Fexamples\u002Fgym\u002Fexample_\u003Ctask>.yaml`. The `gym\u002F` dir groups by **harness** (0.2.6 `nemo_gym`), **not** by suite membership, so read AA membership per task from the table below — never from the path. Every gym task is **standalone**: generated as its own config from its example, one gym eval per config, **never merged into the `aa\u002F` multi-task `tasks` list** and never mixed with each other.\n\n  | Task | Recipe \u002F example | In AA suite? | Generate when |\n  | --- | --- | --- | --- |\n  | **GDPVal** (Stirrup agent, agentic) | `recipes\u002Ftasks\u002Fgym\u002Fgdpval.md` + `references\u002Fgym-gdpval.md`, `recipes\u002Fexamples\u002Fgym\u002Fexample_gdpval.yaml` | **Yes** | any AA request (see the AA rule below) |\n  | **MRCR** (simple agent, long-context) | `recipes\u002Ftasks\u002Fgym\u002Fmrcr.md`, `recipes\u002Fexamples\u002Fgym\u002Fexample_mrcr.yaml` | **No** | only when the user asks for MRCR by name, or for long-context coverage |\n\n**AA rule:** If the user mentions \"AA\" \u002F \"Artificial Analysis\", generate the `recipes\u002Ftasks\u002Faa\u002F` tasks (one multi-task config) **plus a companion standalone GDPVal config** (`recipes\u002Ftasks\u002Fgym\u002Fgdpval.md`, via the GDPVal branch) — GDPVal is part of the AA suite but a different harness, so it's its own config, never added to the `aa\u002F` `tasks` list. Do not add MMLU-Pro, AIME 2025, or LiveCodeBench unless explicitly asked. GDPVal is the heaviest AA task (standalone, multi-hour, needs the SIF sandbox + judge) — surface it and let the user opt out per run.\n\n**Shortcut path** (when task list is known up front, e.g. \"run AA\"):\n\n1. Read the task reference file(s).\n2. Use `recipes\u002Fexamples\u002Fexample_eval.yaml` as the base.\n3. Copy the YAML fragment(s) into `evaluation.tasks`, applying any per-task notes.\n4. **MLflow auto-export is on by default** — it needs **two** pieces, both in `example_eval.yaml`: (a) the **trigger** `execution.auto_export.destinations: [mlflow]` (without it the run is *not* uploaded), and (b) the `export.mlflow` block that configures it. In the `export.mlflow` block use **literal** values for `experiment_name` \u002F `description` \u002F `tags` — substitute the actual `served_model_name` and sampling params. Do **not** use `${deployment.*}` \u002F `${evaluation.*}` cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with `Interpolation key '...' not found` (`${oc.env:USER}` and `${oc.env:MLFLOW_TRACKING_URI}` are fine — they're env vars). Because these literals can't interpolate, keep the `temperature` \u002F `top_p` \u002F `max_new_tokens` tags **equal to** the top-level `params` and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. `tracking_uri` = `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config` (not hand-filled), and auto-export needs `execution.cpu_partition` (e.g. gcp-nrt `cpu`) — it's a separate CPU-only sbatch that GPU-only partitions reject (`Cannot find GPU specification`), silently dropping the link.\n5. Proceed to Step 3, then Step 4, then Step 7.5\u002F8. Skip Step 2's 5-question flow.\n\n---\n\n### Step 2 — Build base config (when not using shortcut)\n\nAsk the 5 questions via AskUserQuestion (categories must match `nel skills build-config --help` — **run that first** to confirm the current option names; CLI options override this list).\n\n1. **Execution:** Local \u002F SLURM\n2. **Deployment:** None (External) \u002F vLLM \u002F SGLang \u002F NIM \u002F TRT-LLM. Prefer vLLM unless the user\u002Fcard says otherwise.\n3. **Auto-export:** None \u002F MLflow \u002F wandb\n4. **Model type:** Base \u002F Chat \u002F Reasoning\n5. **Benchmarks** (multi-select): standard \u002F code \u002F math_reasoning \u002F safety \u002F multilingual\n\nBuild the base:\n\n```bash\nnel skills build-config --execution \u003C...> --deployment \u003C...> --model_type \u003C...> --benchmarks \u003C...> [--export \u003C...>] [--output \u003C...>]\n```\n\n(`--output` omitted = cwd auto-named; directory = dir + auto-name; `*.yaml` = exact path. Never overwrites.)\n\n---\n\n### Step 3 — Configure deployment\n\n**Model path.** Checkpoint path (`\u002F`, `.\u002F`, `..\u002F`, `~`, or exists on disk) → set `deployment.checkpoint_path`, leave `hf_model_handle: null`. Else HF handle (one `\u002F`, not on disk) → set `deployment.hf_model_handle`, leave `checkpoint_path: null`.\n\n> **NEVER point `checkpoint_path` at a HuggingFace *cache snapshot* dir.** Entries under\n> `snapshots\u002F\u003Csha>\u002F` are relative symlinks into `..\u002F..\u002Fblobs\u002F`. NEL mounts only the snapshot dir at\n> `\u002Fcheckpoint`, so every link dangles in-container and vLLM dies with\n> `Invalid repository ID or local directory specified: '\u002Fcheckpoint'`. Pre-staging into `HF_HOME`\n> does not help — that works for `from_pretrained`, not a mounted directory. Build a hardlink farm\n> (same filesystem, no extra space) and point at that:\n>\n> ```bash\n> find \"$SNAP\" ! -type d -exec sh -c '\n>   for f; do\n>     rel=${f#\"$SNAP\"\u002F}; mkdir -p \"$DEST\u002F$(dirname \"$rel\")\"\n>     ln -f \"$(readlink -f \"$f\")\" \"$DEST\u002F$rel\"\n>   done' _ {} +\n> ```\n>\n> (`SNAP`\u002F`DEST` must be **exported** — they are read inside a new shell — and the loop preserves\n> nested paths; `basename` would flatten subdirectories into one level.)\n> The same applies to ModelOpt exports whose `--source_ckpt` was a snapshot: the exporter preserves\n> symlinks, shipping a dangling `tokenizer.json`. Check with `find \"$OUT\" -type l` before serving.\n>\n> **Prefer `checkpoint_path` over `hf_model_handle` on SLURM** — `hf_model_handle` isn't reliably mounted at `\u002Fcheckpoint`, so the deploy dies with `HFValidationError`. To eval an un-staged HF model, stage it first (`huggingface_hub.snapshot_download`) and point `checkpoint_path` at it. See `example_eval.yaml` for why.\n\n**Auto-detect ModelOpt quantization** (checkpoint paths). Check `config.json` for `quantization_config` (or legacy `hf_quant_config.json`):\n\n- **vLLM:** no `--quantization` flag by default — vLLM auto-detects from `quantization_config` \u002F `hf_quant_config.json`. Add only when the card, vLLM version, or dry-run error requires it.\n- **SGLang:** may need `--quantization modelopt_fp8` \u002F `modelopt_fp4` \u002F `modelopt` — verify against installed version.\n\nSome models need extra vLLM backend env vars (model-card research) — e.g. `VLLM_NVFP4_GEMM_BACKEND=marlin` (Nemotron Super), or `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput` (NVFP4 MoE, e.g. NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4). Put them in `deployment.env_vars` (**not** `command`) with the `lit:` prefix (`VLLM_USE_FLASHINFER_MOE_FP4: lit:1`); see `example_eval.yaml` and Step 5's prefix rule.\n\n**Auto-detect from `config.json`:**\n\n| Field | Flag |\n| --- | --- |\n| `max_position_embeddings` | `--max-model-len \u003Cvalue>` |\n| `auto_map` exists | `--trust-remote-code` |\n\n#### Cross-check both sources for vLLM (mandatory, neither replaces the other)\n\n**Source 1 — `recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>`** (curated vLLM recipes; authoritative for parallelism, family-specific flags like `--reasoning-parser` \u002F `--tool-call-parser` \u002F `--mm-encoder-tp-mode`, vLLM version, spec-decoding, GPU count). **Fetch the page for the EXACT model id, not a base\u002Fsibling** — variant minimums differ (e.g. MiniMax-M2 ≥0.11.0 vs M2.7 ≥0.20.0). Pin variants via query params (e.g. `?variant=fp8&strategy=single_node_tep`).\n\n> **WebFetch caveat — triage the summary:**\n>\n> 1. **\"No `vllm serve` commands found\" \u002F \"page is a usage guide\":** JS-rendering miss. recipes.vllm.ai pages always have ≥1 command. Ask the user to paste it or share the variant URL.\n> 2. **Single recipe returned** for a model with known multiple variants → retry with variant-pinned URL. Axis names differ per model (Qwen: `?variant=&strategy=`; Kimi: `?advanced=`; others vary — no fixed pattern).\n> 3. **Variant label contradicts the command** (e.g. label \"TEP\" but command shows DP+EP) → summarizer conflated variants; ask user.\n>\n> For non-trivial deployments (≥120B, multi-node, novel arch), ask the user which variant *before* fetching.\n\n**Source 2 — HF model card + `config.json`** (authoritative for):\n\n| Signal | Flag |\n| --- | --- |\n| `max_position_embeddings` | `--max-model-len \u003Cvalue>` |\n| `auto_map` | `--trust-remote-code` |\n| Reasoning\u002FCoT documented | `--reasoning-parser` (and `--reasoning-parser-plugin` if custom) |\n| Tool-calling documented | `--enable-auto-tool-choice --tool-call-parser \u003Cparser>` |\n| Custom flags in card | Add as specified (e.g. `--mamba_ssm_cache_dtype float32`) |\n\n**Cross-check rules:**\n\n1. Read both sources before composing the command.\n2. Agree → use with confidence.\n3. Disagree → **do not silently pick one.** Surface both values to the user. Common conflicts: stale cards, parser rename between generations (Qwen2.5 `hermes` → Qwen3 `qwen3_coder`), recipe-only flags like `--language-model-only`, ARM64-specific card notes.\n4. Resolve in Step 3 — don't defer to dry-run.\n\n#### vLLM deployment command structure — single `command:` field\n\nRewrite the build-config output into one `command:` field. Move all parallelism (`--tensor-parallel-size`, `--data-parallel-size`, `--pipeline-parallel-size`) into the command; do not keep separate `tensor_parallel_size` \u002F `data_parallel_size` \u002F `extra_args` YAML fields.\n\n```yaml\ndeployment:\n  command: >-\n    vllm serve \u002Fcheckpoint\n    --host 0.0.0.0\n    --port ${deployment.port}\n    --tensor-parallel-size \u003CN>\n    --data-parallel-size \u003CM>\n    --max-model-len \u003Cvalue>\n    \u003C... rest of cross-checked flags ...>\n```\n\nConventions: always start `vllm serve \u002Fcheckpoint` (NEL mounts here); always `--served-model-name ${deployment.served_model_name}` (**required**; see `example_eval.yaml` for why); always `--host 0.0.0.0 --port ${deployment.port}`; use folded scalar (`>-`) for one flag per line. Example fallback `--max-model-len 131072` covers AA-LCR (~120K + 16K gen) and SciCode (≥ 65536) — prefer `config.json` \u002F recipe value.\n\nFor how to choose `--tensor-parallel-size` \u002F `--data-parallel-size` \u002F `--pipeline-parallel-size` (and EP) from the model size and your GPU count, read `references\u002Fparallelism.md` — cross-check the layout against `recipes.vllm.ai`, then adapt to the GPUs you actually have via the fit math there.\n\n**Image \u002F vLLM version.** Treat default `image: vllm\u002Fvllm-openai:v0.26.0` as a floor to verify: bump to the **exact model's** `recipes.vllm.ai` minimum if higher. Running below minimum is a trap — the server starts, then a worker dies mid-inference with `CUDA error: an illegal memory access`, easy to misread as a kernel bug. A model newer than the latest release may have no numbered tag — use the image its recipe names. Never `:latest` (breaks reproducibility). Surface version bumps to the user.\n\n> **NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 build** — the cu12 build has no sm_103 FP4 kernel, so engine init dies with `CUDA error: no kernel image is available`. **Pick the tag by the CUDA version it reports, not by its name — vLLM inverted its tag convention at v0.20.0:**\n>\n> | vLLM version | CUDA-13 tag | CUDA-12 tag |\n> | --- | --- | --- |\n> | ≤ v0.19.x | **suffixed** `-cu130` | unsuffixed |\n> | ≥ v0.20.0 | **unsuffixed** | suffixed `-cu129` |\n>\n> v0.20.0 ships both suffixes; after it `-cu130` doesn't exist, so asking for it yields a missing tag. **Select a tag whose config blob reports `CUDA_VERSION` ≥ 13** (registry API), reading the child manifest for **the platform you deploy on** (arm64 Grace\u002FGB300, amd64 x86) — `TORCH_CUDA_ARCH_LIST` differs per platform, so check your arch against that child. Multimodal on sm_103 may also need `--mm-encoder-attn-backend TRITON_ATTN`. Full note in `recipes\u002Fexamples\u002Fexample_eval.yaml`.\n\n#### vLLM-backend defaults — always include unless the recipe *contradicts*\n\nSilence is not contradiction. Drop\u002Foverride only when the recipe sets a different value for the same setting (e.g. recipe pins `--max-num-batched-tokens 16384` → use 16384).\n\n- `--model-loader-extra-config '{\"enable_multithread_load\": true, \"num_threads\": 128}'` — **parallelizes checkpoint load**, the single biggest deploy-time cost for large checkpoints. A big MoE otherwise loads shards ~sequentially (~1 min\u002Fshard → e.g. ~40 min for a ~450 GB \u002F 45-shard checkpoint); on a **preemptible** queue that long load window is exactly where jobs get killed before they ever serve. `num_threads` defaults to **128**; scale it to the checkpoint (smaller for small models, bounded by the shared-FS read bandwidth — too high yields no gain). Safe to always include.\n- `--max-num-batched-tokens 8192` — caps per-step batched tokens; prevents long-prefill stalls.\n- `--enable-chunked-prefill` — interleaves long prefills with decode steps (required for AA-LCR's ~120K input). Modern vLLM defaults this on for many models; set explicitly to avoid drift.\n- `--enable-expert-parallel` — **MoE-only default.** Detect MoE from handle suffix (`-A10B`, `-A3B`, etc.), `num_experts` \u002F `num_local_experts` \u002F `n_routed_experts` in `config.json`, or card. No-op when TP=DP=1, safe to always include for MoE. Do not add for dense models. See `references\u002Fparallelism.md` for what EP does and the DP-attention + EP-MoE throughput pattern.\n- `--max-num-seqs N` — **omit at generation time** (top-level `parallelism` is `???`). Add this comment above `command:`:\n\n  ```text\n  # After filling in `parallelism` values (top-level + per-task overrides),\n  # append `--max-num-seqs N` where N = ceil(max_parallelism \u002F data_parallel_size).\n  ```\n\n  In Step 4 compute and append. Example: top-level=16, Tau2=128, DP=8 → `ceil(128\u002F8)=16`. Too small → request queuing; too large → wasted KV reservation. For how to choose the `parallelism` it derives from, read `references\u002Fparallelism.md`.\n\n#### Evaluation params template (top-level params)\n\nThe top-level `nemo_evaluator_config.config.params` must contain **exactly these six fields** — no `top_k` \u002F `presence_penalty` \u002F `repetition_penalty` \u002F `min_p`:\n\n```yaml\nnemo_evaluator_config:\n  config:\n    params:\n      parallelism: ???    # Required — size per references\u002Fparallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear\n      request_timeout: 3600\n      max_retries: 10\n      max_new_tokens: 65536  # see rule below\n      temperature: 1.0    # from model card (reasoning); adjust\n      top_p: 0.95         # from model card (reasoning); adjust\n```\n\nPer-task `max_new_tokens` overrides are forbidden — set one top-level ceiling everywhere.\n\n**Cross-check `temperature` \u002F `top_p` \u002F `max_new_tokens` against `references\u002Fnvfp4-modelcard-sampling.md`** — the published settings for the 2026 NVFP4 checkpoints under `huggingface.co\u002Fnvidia` that disclose them (older releases and cards that publish nothing are absent — for those, read the card; `-DSpark` \u002F `-DFlash` spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). **The card is the source of truth; this file is a reference, not a constraint** — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint **or shares a family with one** (Qwen3.x, GLM-4.7\u002F5.x, Kimi K2.x\u002FK3, MiniMax M2.x\u002FM3, DeepSeek V3.x\u002FV4\u002FR1, Gemma 4, Nemotron 3\u002F3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's \"Lookup\" section.\n\n**`temperature` \u002F `top_p` are different: per-task overrides ARE allowed and often required.** Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives `top_p = 0.95` for agentic scenarios and `1.0` otherwise, so a single top-level `0.95` is wrong for every non-agentic task.\nSet the top-level value for the majority case, override only the tasks the card calls out, and apply\nthe split identically to baseline and candidate. **The `export.mlflow` tags record only the\ntop-level values**, so note any per-task override in the run `description` — otherwise the\noverridden task is reported under sampling params it did not use.\n\n#### `max_new_tokens` — mandatory model-card lookup\n\n1. **Fetch the HF model card before writing the value.** Not optional.\n2. Scan for any `max_tokens` \u002F `max_new_tokens` \u002F \"output length\" recommendation. Pick the **highest** value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use **81920**). Annotate with a citing comment.\n   **Card figures are SINGLE-TURN.** On multi-turn \u002F agentic benchmarks the model's own answer is fed back in, so the cap must satisfy `n_turns × max_new_tokens + prompt \u003C max_model_len`. Taking a card's headline \"384K output\" literally lost SciCode samples to HTTP 400; 65536 was clean. (`references\u002Frun-validation.md` already covers checking `finish_reason: length` after a run.)\n3. **Consult `references\u002Fnvfp4-modelcard-sampling.md` as a reference.** Listed and in agreement → proceed with confidence. Listed and different → **the card wins**; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its `max_num_tokens` column records the card's *headline* cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row.\n4. If the card is genuinely silent after a thorough read **and** the family table offers no usable pattern, fall back to: **65536** (reasoning), **16384** (non-reasoning); surface the silence to the user.\n5. **Forbidden:** writing `max_new_tokens: \u003Cgeneric_default>` with a \"card not yet checked\" comment. Either fetch and apply, or fetch and confirm silence.\n6. **A higher cap doesn't fix runaway reasoning.** On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment `--max-model-len > prompt + max_new_tokens` (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence.\n\n#### Quantization-aware benchmark defaults\n\nFor quantized checkpoints, read `references\u002Fquantization-benchmarks.md` for sensitivity rankings and recommended sets; present and ask which to include. Read `references\u002Fmodel-card-research.md` for the full extraction checklist (sampling, reasoning config, ARM64, `pre_cmd`, output length — see the dedicated bullet there).\n\nReasoning models: prefer reasoning mode (highest scores). For lower variance \u002F cost \u002F apples-to-apples vs non-reasoning baselines, also consider a non-reasoning companion run.\n\n#### Reasoning adapter config (`use_reasoning`)\n\nThe `adapter_config` block in `example_eval.yaml` controls request\u002Fresponse\nlogging and reasoning handling. `use_reasoning: true` strips the model's\nreasoning\u002FCoT trace before scoring (grade only the final answer). Set per type:\n\n1. **Instruct → `use_reasoning: false`** and drop the `chat_template_kwargs`\n   thinking block (no trace to strip; can mangle plain responses).\n2. **Reasoning → `use_reasoning: true`**, especially when the deployment sets\n   `--reasoning-parser` (vLLM emits a separate reasoning channel to strip).\n3. **Hybrid (reasoning on *or* off) → turn it ON** (`use_reasoning: true` +\n   force the thinking flag in `chat_template_kwargs`). For the exact toggle key\n   (it drifts across generations) and the reasoning-effort policy, see\n   `references\u002Fmodel-card-research.md` → \"Reasoning config\".\n\n---\n\n### Step 4 — Fill remaining ??? values\n\n**Predefined per-cluster execution config (check FIRST).** Some installs ship `internal\u002Fslurm\u002F\u003Ccluster>` execution groups (optional `nemo_evaluator_launcher_internal` pkg) that pre-fill hostname\u002Fpartition\u002Fgres — leaving only account\u002Foutput_dir\u002Fwalltime. Discover at runtime (nothing cluster-specific hardcoded):\n\n```bash\npython3 -c 'import nemo_evaluator_launcher_internal' 2>\u002Fdev\u002Fnull && \\\nPKG=$(python3 -c 'import nemo_evaluator_launcher_internal as m,os;print(os.path.dirname(m.__file__))') && \\\nfor f in \"$PKG\"\u002Fconfigs\u002Fexecution\u002Finternal\u002Fslurm\u002F*.yaml; do \\\n  echo \"$(basename \"$f\" .yaml) -> $(grep -E '^hostname:' \"$f\" | awk '{print $2}')\"; done\n```\n\nHostname match → set `defaults: - execution: internal\u002Fslurm\u002F\u003Ccluster>`, drop the redundant `execution.hostname` (keep account\u002Foutput_dir\u002Fwalltime), verify with `--dry-run`. Else keep `slurm\u002Fdefault` and fill hostname\u002Faccount\u002Foutput_dir manually.\n\nOn SLURM, several deploy\u002Feval failures are invisible to `--dry-run` and only surface at canary (`mount_home`, HF cache, `cpu_partition`, top-level vs per-stage `env_vars`) — read `references\u002Fslurm.md`.\n\n- Find every `???` left. Ask the user only for what can't be inferred (SLURM hostname\u002Faccount\u002Foutput_dir, the `cpu_partition` for auto-export, etc.). Don't propose defaults; let them give plain text. (`tracking_uri` is **not** one of these — it's `${oc.env:MLFLOW_TRACKING_URI}` from `modelopttools:eval-config`.)\n- **`parallelism`** — size it yourself from the run shape (total requests = `dataset_size × repeats` vs GPU serving capacity), and set `--max-num-seqs` to match. Read `references\u002Fparallelism.md` for the decision rule and worked examples; only ask the user if a non-GPU cap (e.g. judge rate limit) is unknown.\n- Ask about other defaults they may want to change (partition, walltime, MLflow tags).\n- **`execution.gres`** — auto-set if you used a predefined `internal\u002Fslurm\u002F\u003Ccluster>` config (above). On the `slurm\u002Fdefault` fallback it's `gpu:8`, so set it to the node's GPU count (and match `--data-parallel-size`\u002F`--tensor-parallel-size`) or `sbatch` rejects the job with *\"Requested node configuration is not available\"* (e.g. 4-GPU GB300 → `gres: gpu:4`; check with `sinfo -o '%P %G'`).\n\n**Walltime cap: 4 hours.** Always `execution.walltime: \"04:00:00\"`. The cluster does not schedule jobs longer than 4h — this is a hard limit, not a preference.\n\nEvals that exceed 4h of wall-clock time are handled by **NEL's built-in dependency-chain resume**, not by shrinking the eval. NEL submits the first SLURM job; if it hits walltime, a dependent follow-on job resumes from the response\u002Fresult caches the first job wrote, then queues another follow-on. Long evals continue across walltime windows automatically. See `references\u002Frun-validation.md#nel-timeout-and-resume-behavior` for the full mechanism.\n\n**Never `scancel` a wedged run to free the GPUs.** The dependency-chain resume above fires only on a genuine walltime timeout; `afternotok` explicitly refuses a predecessor that finished `CANCELLED by \u003Cuid>` and exits in seconds, discarding the response cache the run had already filled. Let it hit the wall clock instead — one `scancel` threw away ~59 GPU-h of completed generation.\n\n**Preemption \u002F external kill — resume manually with `sbatch run.sub`.** On a preemptible account (common on busy internal clusters) the scheduler can **CANCEL** a run mid-eval for a higher-priority job — `sacct -j \u003Cid>` shows `CANCELLED by \u003Cuid>` (a `svc-*` service account) with `Elapsed` well under the 4h walltime. NEL does **not** auto-resume this (its dependency chain only fires on a genuine walltime timeout). But the `run.sub` that NEL generated for the job (in its run dir) is **re-submittable** and resumes from the same `output_dir` + response cache (`skip_filled`), continuing from the partial output rather than restarting:\n\n```bash\nssh \u003Chost> \"cd \u003Coutput_dir>\u002F\u003Ctimestamp>-\u003Cinvocation>\u002F\u003Ctask>\u002F && sbatch run.sub\"\n```\n\nRe-submit again if it's preempted again — each resume re-deploys, then skips already-generated samples, so progress is **cumulative** across attempts until it completes. Always confirm via `sacct -j \u003Cid>` that the prior job was `CANCELLED` (not a real failure) before resuming.\n\nImplications for the agent:\n\n- Do **not** lower `num_repeats`, split heavy tasks (AA-LCR, SciCode) into separate configs, or otherwise carve up the eval to fit inside 4h. Let NEL chain.\n- Do **not** treat a walltime timeout as a failed run. Check `nel status` \u002F `nel info` and the dependent job's logs before declaring failure. `references\u002Frun-validation.md` covers what a real failure looks like vs an expected resume event.\n- Bumping `data_parallel_size` \u002F `parallelism` to finish faster is fine when the goal is wall-clock latency, not a walltime workaround — but it's optional, not required, for runs longer than 4h.\n\n---\n\n### Step 5 — Confirm tasks (iterative)\n\n1. Tell user: \"Run `nel ls tasks` for the full task list.\"\n2. For any task with a `recipes\u002Ftasks\u002F` reference, read it and prefer its YAML fragment + repeat counts.\n3. Ask about add\u002Fremove\u002Fmodify. Per-task overrides under task's `nemo_evaluator_config.config.params`:\n\n   ```yaml\n   tasks:\n     - name: \u003Ctask>\n       nemo_evaluator_config:\n         config:\n           params:\n             temperature: \u003Cvalue>\n             ...\n   ```\n\n4. Apply, show updated list, ask \"Final, or more changes?\" Loop until confirmed.\n\n**Tasks that call an external judge \u002F user-simulator \u002F scoring endpoint.** Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their `model_id` \u002F `url` are **config, not secrets**: substitute the **literal** values the user keeps in `.env` (keys per the task's recipe + `recipes\u002Fenv.example`) into the task's `\u003CVAR>` placeholders. Do **not** emit `${oc.env:...}` for these (it silently fails unless the var was exported with `set -a`). Only `api_key` stays an env-var *name* (e.g. `INFERENCE_API_KEY`), exported and read by the harness. All judges + user-sims (HLE, AA-LCR, Tau2, AIME) use one `INFERENCE_API_KEY` against one OpenAI-compatible host (AIME's simple-evals default judge endpoint is overridden to it — see its recipe). Get the `*_MODEL_ID`\u002F`*_URL` values via `modelopttools:eval-config` (see Step 1) rather than guessing a host; only fill them by hand if it's unavailable. `.env` should already exist (Step 1) — if not, set it up now (don't defer to Step 8) before substituting.\n\n**Known issue — nemo-skills self-deployment:** If using `nemo_skills.*` tasks (`ns_*`) with self-deployment (vLLM\u002FSGLang\u002FNIM), you need **both** of these:\n\n```yaml\nevaluation:\n  env_vars:\n    DUMMY_API_KEY: lit:dummy   # MUST be set here — see below\n  nemo_evaluator_config:\n    target:\n      api_endpoint:\n        api_key_name: DUMMY_API_KEY\n```\n\n`api_key_name` only names the env var; the nemo-skills client **hard-fails if that var has no value inside the eval container** (`ValueError: api_key_env_var=DUMMY_API_KEY but the value is not set`). On SLURM, a shell `export DUMMY_API_KEY=dummy` (Step 8) does **NOT** propagate into the container — NEL only injects vars declared in `env_vars`. So declare `DUMMY_API_KEY: lit:dummy` under `evaluation.env_vars` (note the `lit:` prefix — see below). The shell export only helps for local\u002FDocker runs. External-deployment configs already define `api_key_name`.\n\n**NEL env-var value prefixes (required):** every value in an `env_vars` map needs an explicit prefix — `host:VAR` (read from the submitting shell's env at submit time), `lit:value` (literal string), or `runtime:VAR` (read in the job at run time). A bare value (e.g. `DUMMY_API_KEY: dummy`) hard-errors: *\"Env var value '…' must have an explicit prefix.\"* Use `lit:` for constants like `DUMMY_API_KEY` and `VLLM_*` backend selectors, `host:` for secrets like `HF_TOKEN` \u002F `INFERENCE_API_KEY`.\n\n---\n\n### Step 6 — Multi-node\n\nFor models > ~120B or higher throughput needs, read `references\u002Fmulti-node.md` for HAProxy multi-instance \u002F Ray TP\u002FPP \u002F combined patterns.\n\n### Step 7 — Interceptors\n\nDirect user to \u003Chttps:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fevaluator\u002Flatest\u002Flibraries\u002Fnemo-evaluator\u002Finterceptors\u002Findex.html>. Do not provide generic interceptor info — read the specific interceptor's page if asked, then configure via `evaluation.nemo_evaluator_config.target.api_endpoint.adapter_config` (`target` is a sibling of `config`, not nested under it). Use the per-field syntax from the CLI Configuration section, not a full `interceptors:` list (that overrides the default chain).\n\n**Errata:** Logging field names are `max_logged_requests` \u002F `max_logged_responses` (NOT `max_saved_*` \u002F `max_*` as some docs show).\n\n### Step 7.5 — Container registry auth (SLURM private images only)\n\nDefault images:\n\n| Framework | Image | Registry |\n| --- | --- | --- |\n| vLLM | `vllm\u002Fvllm-openai:v0.26.0` (bump per recipe; never `:latest`) | DockerHub |\n| vLLM (NVFP4 on B300\u002FGB300) | default is already **CUDA-13**; for older pins see Step 3 | DockerHub |\n| SGLang | `lmsysorg\u002Fsglang:latest` | DockerHub |\n| TRT-LLM | `nvcr.io\u002Fnvidia\u002Ftensorrt-llm\u002Frelease:...` | NGC |\n| Eval tasks | `nvcr.io\u002Fnvidia\u002Feval-factory\u002F*:26.03` | NGC |\n\n> NVFP4 checkpoints on B300\u002FGB300 (sm_103) need a **CUDA-13** image — CUDA-12 builds lack sm_103 FP4 kernels. The tag spelling depends on the vLLM version (Step 3 table); verify `CUDA_VERSION` in your platform's child manifest.\n\nPublic images → submit without preflight. Private\u002Frestricted → check credentials:\n\n```bash\nssh \u003Chost> \"grep -E '^\\s*machine\\s+' ~\u002F.config\u002Fenroot\u002F.credentials 2>\u002Fdev\u002Fnull\"\n```\n\nAdd credentials per the common skill's `slurm-setup.md` §6 if missing. If you can't add, switch to a compatible public image (e.g. `nvcr.io\u002Fnvidia\u002Fvllm:\u003CYY.MM>-py3` — check catalog.ngc.nvidia.com). **Do not retry more than once** after an auth failure.\n\n---\n\n### Step 8 — Run evaluation (gated dry-run → canary → full)\n\nRun directly when the user asked to launch; otherwise ask before submitting.\n\n**Env setup:** `.env` is normally already created and filled back in Step 1 (via `modelopttools:eval-config`), at the **workspace root** — the dir you run `nel` from, not under the skill dir. Ensure it exists and source it — do **not** clobber an existing `.env`:\n\n```bash\n# .env lives at the workspace root (where you run nel); the template ships under the skill dir\n[ -f .env ] || cp \"$SKILL_DIR\u002Frecipes\u002Fenv.example\" .env   # create only if Step 1 didn't\nset -a && source .env && set +a\n\n# If pre_cmd\u002Fpost_cmd in config (review pre_cmd first — runs arbitrary commands):\nexport NEMO_EVALUATOR_TRUST_PRE_CMD=1\n# If nemo_skills.* + self-deployment, for LOCAL\u002FDocker runs only:\nexport DUMMY_API_KEY=dummy\n# On SLURM this shell export does NOT reach the container — instead declare\n# `DUMMY_API_KEY: lit:dummy` under evaluation.env_vars (see Step 5).\n```\n\n**Step 8.1 — Dry-run** (config validation):\n\n```bash\nnel run --config \u003Cpath> --dry-run\n```\n\nFix unresolved `???`, bad Hydra overrides, missing env vars, invalid mounts, image issues, sbatch errors, obvious deployment errors before proceeding.\n\n> **Dry-run does NOT validate the image\u002FvLLM version** (image pulled only at deploy). Confirm `image:` ≥ the exact model's `recipes.vllm.ai` minimum (Step 3) before submitting — too-old passes dry-run, then crashes mid-inference.\n\n> **Non-fatal noise:** \"Failed to get manifest\"\u002F`401`\u002F`404`, \"Could not extract frame definition file\", \"proceeding with minimal task definition\", \"Found N unlisted task(s)\" — expected for `ns_*`\u002Frecipe tasks and private (gitlab) containers; the task still runs in-container. Set `NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1`. Real blockers: unresolved `???`, interpolation errors, bad mounts, sbatch rejections.\n\n**Step 8.2 — Canary** (limited-samples, validates everything dry-run can't):\n\n```bash\nnel run --config \u003Cpath> -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10\n```\n\nCatches judge auth\u002Frate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. Always inspect logs:\n\n```bash\nnel status \u003Cid>\nnel info \u003Cid> --logs\nssh \u003Cuser>@\u003Chost> \"grep -i 'traceback\\|exception\\|error\\|failed\\|oom\\|killed\\|timeout\\|unauthorized\\|rate limit\\|sandbox\\|container\\|judge\\|parse\\|scoring' \u003Clog_path>\u002F*.log\"\n```\n\nCanary each risky task class separately (judge-scored, code-execution, model-only). Start `parallelism` conservatively; raise only after judge\u002Fsandbox logs are clean — they bottleneck before the model. For capacity-bound runs, tune `parallelism`\u002F`--max-num-seqs` here against vLLM's reported max concurrency + preemption — see `references\u002Fparallelism.md`.\n\nSingle-task rerun: `nel run --config \u003Cpath> -t \u003Ctask_name>` (combine with `-o ++...limit_samples=10` for canary).\n\n**Step 8.3 — Full run** (after canary passes):\n\n```bash\nnel run --config \u003Cpath>\n```\n\nRemove `limit_samples` overrides; keep canary-validated parallelism. If the canary fails, fix and rerun the canary — don't skip to full.\n\n**Monitoring:** Register the job per the **monitor** skill for cross-session tracking. One-off live status \u002F debugging → **launching-evals** skill. Past-run MLflow queries → **accessing-mlflow** skill. NEL timeout\u002Fresume → read `references\u002Frun-validation.md` before treating the run as failed.\n\n---\n\n### Step 9 — Verify completed run\n\nBefore pulling\u002Freporting scores, validate the run. Read `references\u002Frun-validation.md` for NEL timeout\u002Fresume behavior, completed-run validation, diagnostics, and score harvesting. For a baseline that will be compared with a candidate, also perform its **External Baseline Sanity Check** before a success verdict, then hand the validated runs to `compare-results` for baseline-vs-candidate deltas.\n\n---\n\nIssues: \u003Chttps:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Fissues> · \u003Chttps:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Fdiscussions>\n",{"data":35,"body":36},{"name":4,"description":6,"license":26},{"type":37,"children":38},"root",[39,48,54,61,91,97,109,113,119,222,362,367,370,376,418,636,639,653,698,941,944,950,1016,1082,1109,1258,1318,1328,1643,1687,1697,1958,1961,1967,1986,2039,2044,2217,2238,2241,2247,2322,2644,2678,2744,2815,2830,2896,2903,2955,3035,3050,3173,3181,3235,3249,3305,3412,3477,3517,3564,3726,3737,3750,3954,3960,4008,4175,4187,4259,4325,4336,4513,4519,4546,4551,4564,4591,4677,4680,4686,4712,4979,5016,5059,5226,5244,5264,5305,5397,5444,5471,5476,5548,5551,5557,5703,5841,5873,5978,6054,6151,6154,6160,6173,6179,6224,6264,6270,6275,6420,6441,6446,6489,6517,6520,6526,6531,6579,6780,6790,6837,6849,6877,6926,6936,6989,6994,7124,7155,7176,7186,7224,7236,7274,7277,7283,7310,7313,7332],{"type":40,"tag":41,"props":42,"children":44},"element","h2",{"id":43},"nemo-evaluator-launcher-assistant",[45],{"type":46,"value":47},"text","NeMo Evaluator Launcher Assistant",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52],{"type":46,"value":53},"Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress.",{"type":40,"tag":55,"props":56,"children":58},"h3",{"id":57},"workspace-integration",[59],{"type":46,"value":60},"Workspace integration",{"type":40,"tag":49,"props":62,"children":63},{},[64,66,73,75,81,83,89],{"type":46,"value":65},"If ",{"type":40,"tag":67,"props":68,"children":70},"code",{"className":69},[],[71],{"type":46,"value":72},"MODELOPT_WORKSPACE_ROOT",{"type":46,"value":74}," is set, use the common skill's ",{"type":40,"tag":67,"props":76,"children":78},{"className":77},[],[79],{"type":46,"value":80},"workspace-management.md",{"type":46,"value":82}," and reuse existing workspaces (this skill is usually the final stage of PTQ → Deploy → Eval; carry any deployment-time patches into ",{"type":40,"tag":67,"props":84,"children":86},{"className":85},[],[87],{"type":46,"value":88},"deployment.command",{"type":46,"value":90},").",{"type":40,"tag":55,"props":92,"children":94},{"id":93},"workflow",[95],{"type":46,"value":96},"Workflow",{"type":40,"tag":98,"props":99,"children":104},"pre",{"className":100,"code":102,"language":46,"meta":103},[101],"language-text","- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)\n- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)\n- [ ] Step 2: Build base config (5-question flow OR shortcut)\n- [ ] Step 3: Configure deployment (model path, params, cross-check)\n- [ ] Step 4: Fill remaining ??? values\n- [ ] Step 5: Confirm tasks (iterative)\n- [ ] Step 6: Multi-node (if needed)\n- [ ] Step 7: Interceptors (if needed)\n- [ ] Step 7.5: Container auth (SLURM private images)\n- [ ] Step 8: Dry-run → canary → full run\n- [ ] Step 9: Verify completed run\n","",[105],{"type":40,"tag":67,"props":106,"children":107},{"__ignoreMap":103},[108],{"type":46,"value":102},{"type":40,"tag":110,"props":111,"children":112},"hr",{},[],{"type":40,"tag":55,"props":114,"children":116},{"id":115},"nel-next-path-terminal-bench-2x-swe-bench-branch-here-first",[117],{"type":46,"value":118},"nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST",{"type":40,"tag":49,"props":120,"children":121},{},[122,124,130,132,137,139,145,147,152,154,160,162,168,170,176,178,184,186,192,193,199,200,206,208,212,214,220],{"type":46,"value":123},"A few ",{"type":40,"tag":125,"props":126,"children":127},"strong",{},[128],{"type":46,"value":129},"agentic",{"type":46,"value":131}," AA benchmarks do ",{"type":40,"tag":125,"props":133,"children":134},{},[135],{"type":46,"value":136},"not",{"type":46,"value":138}," run on the currently validated\n",{"type":40,"tag":67,"props":140,"children":142},{"className":141},[],[143],{"type":46,"value":144},"nemo-evaluator-launcher",{"type":46,"value":146}," 0.2.6 path (Steps 1–9 don't apply). They run on ",{"type":40,"tag":125,"props":148,"children":149},{},[150],{"type":46,"value":151},"nel-next",{"type":46,"value":153},"\n(",{"type":40,"tag":67,"props":155,"children":157},{"className":156},[],[158],{"type":46,"value":159},"nemo-evaluator[harbor]",{"type":46,"value":161}," 0.4.x) — a separate package, CLI (",{"type":40,"tag":67,"props":163,"children":165},{"className":164},[],[166],{"type":46,"value":167},"nel eval run",{"type":46,"value":169},"), ",{"type":40,"tag":67,"props":171,"children":173},{"className":172},[],[174],{"type":46,"value":175},"-O",{"type":46,"value":177},"\noverrides, and ",{"type":40,"tag":67,"props":179,"children":181},{"className":180},[],[182],{"type":46,"value":183},"services",{"type":46,"value":185},"\u002F",{"type":40,"tag":67,"props":187,"children":189},{"className":188},[],[190],{"type":46,"value":191},"benchmarks",{"type":46,"value":185},{"type":40,"tag":67,"props":194,"children":196},{"className":195},[],[197],{"type":46,"value":198},"cluster",{"type":46,"value":185},{"type":40,"tag":67,"props":201,"children":203},{"className":202},[],[204],{"type":46,"value":205},"output",{"type":46,"value":207}," schema. If the user asks\nfor one, do ",{"type":40,"tag":125,"props":209,"children":210},{},[211],{"type":46,"value":136},{"type":46,"value":213}," add it to a 0.2.6 ",{"type":40,"tag":67,"props":215,"children":217},{"className":216},[],[218],{"type":46,"value":219},"evaluation.tasks",{"type":46,"value":221}," list — instead:",{"type":40,"tag":223,"props":224,"children":225},"ol",{},[226,259,280,328],{"type":40,"tag":227,"props":228,"children":229},"li",{},[230,232,241,243,249,251,257],{"type":46,"value":231},"Read ",{"type":40,"tag":125,"props":233,"children":234},{},[235],{"type":40,"tag":67,"props":236,"children":238},{"className":237},[],[239],{"type":46,"value":240},"references\u002Fnel-next.md",{"type":46,"value":242}," (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe ",{"type":40,"tag":67,"props":244,"children":246},{"className":245},[],[247],{"type":46,"value":248},"recipes\u002Ftasks\u002Faa_next\u002F{terminal_bench_2_1,swebench_verified}.md",{"type":46,"value":250},"; start from ",{"type":40,"tag":67,"props":252,"children":254},{"className":253},[],[255],{"type":46,"value":256},"recipes\u002Fexamples\u002Fexample_eval_next.yaml",{"type":46,"value":258},".",{"type":40,"tag":227,"props":260,"children":261},{},[262,264,270,272,278],{"type":46,"value":263},"Isolated nel-next venv: ",{"type":40,"tag":67,"props":265,"children":267},{"className":266},[],[268],{"type":46,"value":269},"\"$SKILL_DIR\u002Fscripts\u002Fnel-next.sh\" --setup-only",{"type":46,"value":271}," (keeps 0.2.6 ",{"type":40,"tag":67,"props":273,"children":275},{"className":274},[],[276],{"type":46,"value":277},"nel",{"type":46,"value":279}," untouched).",{"type":40,"tag":227,"props":281,"children":282},{},[283,285,294,296,302,304,310,312,318,320,326],{"type":46,"value":284},"Run ",{"type":40,"tag":125,"props":286,"children":287},{},[288],{"type":40,"tag":67,"props":289,"children":291},{"className":290},[],[292],{"type":46,"value":293},"modelopttools:eval-config",{"type":46,"value":295}," (Step 3b) to write the AWS-sandbox creds + harbor infra rows (",{"type":40,"tag":67,"props":297,"children":299},{"className":298},[],[300],{"type":46,"value":301},"${NEL_NEXT_EVAL_IMAGE}",{"type":46,"value":303},", ",{"type":40,"tag":67,"props":305,"children":307},{"className":306},[],[308],{"type":46,"value":309},"${HARBOR_*_ECR_REPOSITORY}",{"type":46,"value":311},") into ",{"type":40,"tag":67,"props":313,"children":315},{"className":314},[],[316],{"type":46,"value":317},".env",{"type":46,"value":319},"; always include the ",{"type":40,"tag":67,"props":321,"children":323},{"className":322},[],[324],{"type":46,"value":325},"output.export_config.mlflow",{"type":46,"value":327}," block.",{"type":40,"tag":227,"props":329,"children":330},{},[331,333,339,341,346,348,354,356,361],{"type":46,"value":332},"Dry-run → canary → full (",{"type":40,"tag":67,"props":334,"children":336},{"className":335},[],[337],{"type":46,"value":338},"nel-next.sh eval run",{"type":46,"value":340},"), then ",{"type":40,"tag":125,"props":342,"children":343},{},[344],{"type":46,"value":345},"push to MLflow",{"type":46,"value":347}," — SLURM doesn't auto-export, so run ",{"type":40,"tag":67,"props":349,"children":351},{"className":350},[],[352],{"type":46,"value":353},"nel-next.sh mlflow-push -r \u003Crun_id> -c \u003Ccfg>",{"type":46,"value":355}," after (config-driven; see ",{"type":40,"tag":67,"props":357,"children":359},{"className":358},[],[360],{"type":46,"value":240},{"type":46,"value":90},{"type":40,"tag":49,"props":363,"children":364},{},[365],{"type":46,"value":366},"Steps 1–9 below are currently validated with 0.2.6 — use them for everything else.",{"type":40,"tag":110,"props":368,"children":369},{},[],{"type":40,"tag":55,"props":371,"children":373},{"id":372},"gdpval-nemo-gym-stirrup-agent-path-branch-here-too",[374],{"type":46,"value":375},"GDPVal (NeMo Gym \"Stirrup\" agent) path — branch here too",{"type":40,"tag":49,"props":377,"children":378},{},[379,381,386,388,393,395,401,403,408,410,416],{"type":46,"value":380},"GDPVal ",{"type":40,"tag":125,"props":382,"children":383},{},[384],{"type":46,"value":385},"does",{"type":46,"value":387}," run on the currently validated 0.2.6 ",{"type":40,"tag":67,"props":389,"children":391},{"className":390},[],[392],{"type":46,"value":277},{"type":46,"value":394}," launcher (as a\n",{"type":40,"tag":67,"props":396,"children":398},{"className":397},[],[399],{"type":46,"value":400},"nemo_gym",{"type":46,"value":402}," task, not nel-next), so Steps 1–9 apply — but it is mechanically\nspecial and ",{"type":40,"tag":125,"props":404,"children":405},{},[406],{"type":46,"value":407},"standalone",{"type":46,"value":409}," (one gym eval per config; never mix it with ",{"type":40,"tag":67,"props":411,"children":413},{"className":412},[],[414],{"type":46,"value":415},"aa\u002F",{"type":46,"value":417},"\ntasks). If the user asks for GDPVal:",{"type":40,"tag":223,"props":419,"children":420},{},[421,446,462,595],{"type":40,"tag":227,"props":422,"children":423},{},[424,425,434,436,445],{"type":46,"value":231},{"type":40,"tag":125,"props":426,"children":427},{},[428],{"type":40,"tag":67,"props":429,"children":431},{"className":430},[],[432],{"type":46,"value":433},"references\u002Fgym-gdpval.md",{"type":46,"value":435}," (Apptainer SIF sandbox, gym prepare\u002Freap\nmachinery, deploy sizing, rubric-vs-comparison scoring, MLflow deliverables trap,\nfailure modes) + ",{"type":40,"tag":125,"props":437,"children":438},{},[439],{"type":40,"tag":67,"props":440,"children":442},{"className":441},[],[443],{"type":46,"value":444},"recipes\u002Ftasks\u002Fgym\u002Fgdpval.md",{"type":46,"value":258},{"type":40,"tag":227,"props":447,"children":448},{},[449,451,460],{"type":46,"value":450},"Start from ",{"type":40,"tag":125,"props":452,"children":453},{},[454],{"type":40,"tag":67,"props":455,"children":457},{"className":456},[],[458],{"type":46,"value":459},"recipes\u002Fexamples\u002Fgym\u002Fexample_gdpval.yaml",{"type":46,"value":461}," — a single\nself-contained file.",{"type":40,"tag":227,"props":463,"children":464},{},[465,467,472,474,479,481,487,489,494,496,502,504,510,512,518,520,526,528,534,536,541,543,549,550,556,557,563,565,571,572,577,579,585,587,593],{"type":46,"value":466},"Prerequisite — the Apptainer SIF. ",{"type":40,"tag":125,"props":468,"children":469},{},[470],{"type":46,"value":471},"If your site provides one, use it",{"type":46,"value":473},"\n(NVIDIA-internal: ",{"type":40,"tag":67,"props":475,"children":477},{"className":476},[],[478],{"type":46,"value":293},{"type":46,"value":480}," Step 3c); otherwise set\n",{"type":40,"tag":67,"props":482,"children":484},{"className":483},[],[485],{"type":46,"value":486},"GDPVAL_SIF_DIR",{"type":46,"value":488}," in ",{"type":40,"tag":67,"props":490,"children":492},{"className":491},[],[493],{"type":46,"value":317},{"type":46,"value":495}," and build with ",{"type":40,"tag":67,"props":497,"children":499},{"className":498},[],[500],{"type":46,"value":501},"\"$SKILL_DIR\u002Fscripts\u002Fgdpval-sif.sh\"",{"type":46,"value":503},"\n(build-if-absent, no cross-cluster copy). Either way the mounted dir must contain\nthe file ",{"type":40,"tag":67,"props":505,"children":507},{"className":506},[],[508],{"type":46,"value":509},"GDPVAL_CONTAINER_PATH",{"type":46,"value":511}," names (template: ",{"type":40,"tag":67,"props":513,"children":515},{"className":514},[],[516],{"type":46,"value":517},"python-3.13.gdpval.sif",{"type":46,"value":519},") — a\nname mismatch passes NEL's ",{"type":40,"tag":67,"props":521,"children":523},{"className":522},[],[524],{"type":46,"value":525},"test -d",{"type":46,"value":527}," check and the agent then silently runs\nunsandboxed. Verify with ",{"type":40,"tag":67,"props":529,"children":531},{"className":530},[],[532],{"type":46,"value":533},"gdpval-sif.sh --check",{"type":46,"value":535},". ",{"type":40,"tag":67,"props":537,"children":539},{"className":538},[],[540],{"type":46,"value":317},{"type":46,"value":542}," needs ",{"type":40,"tag":67,"props":544,"children":546},{"className":545},[],[547],{"type":46,"value":548},"HF_TOKEN",{"type":46,"value":303},{"type":40,"tag":67,"props":551,"children":553},{"className":552},[],[554],{"type":46,"value":555},"INFERENCE_API_KEY",{"type":46,"value":303},{"type":40,"tag":67,"props":558,"children":560},{"className":559},[],[561],{"type":46,"value":562},"TAVILY_API_KEY",{"type":46,"value":564},",\n",{"type":40,"tag":67,"props":566,"children":568},{"className":567},[],[569],{"type":46,"value":570},"INFERENCE_JUDGE_URL",{"type":46,"value":303},{"type":40,"tag":67,"props":573,"children":575},{"className":574},[],[576],{"type":46,"value":486},{"type":46,"value":578},", and ",{"type":40,"tag":67,"props":580,"children":582},{"className":581},[],[583],{"type":46,"value":584},"NEMO_EVALUATOR_TRUST_PRE_CMD=1",{"type":46,"value":586}," (the\nconfig has a ",{"type":40,"tag":67,"props":588,"children":590},{"className":589},[],[591],{"type":46,"value":592},"pre_cmd",{"type":46,"value":594},"). Thinking mode is mandatory (non-thinking loses ~86%).",{"type":40,"tag":227,"props":596,"children":597},{},[598,600,606,608,613,615,621,623,634],{"type":46,"value":599},"Run both dry-run and launch through ",{"type":40,"tag":67,"props":601,"children":603},{"className":602},[],[604],{"type":46,"value":605},"\"$SKILL_DIR\u002Fscripts\u002Fnel-gdpval.sh\"",{"type":46,"value":607},"; it\nenforces the currently validated 0.2.6 launcher even if ",{"type":40,"tag":67,"props":609,"children":611},{"className":610},[],[612],{"type":46,"value":277},{"type":46,"value":614}," on PATH is stale\nand avoids an unset ",{"type":40,"tag":67,"props":616,"children":618},{"className":617},[],[619],{"type":46,"value":620},"NEL_INVOCATION_ID",{"type":46,"value":622}," failure before client startup.\n",{"type":40,"tag":125,"props":624,"children":625},{},[626,632],{"type":40,"tag":67,"props":627,"children":629},{"className":628},[],[630],{"type":46,"value":631},"limit_samples",{"type":46,"value":633}," is inert on the gym path",{"type":46,"value":635}," (the gym runs all 220 tasks\nregardless), so there is no cheap canary: watch the real run's first\n~20–30 min for the SIF-sandbox line and judge auth, and cancel if wrong. See the\nrecipe's Canary section.",{"type":40,"tag":110,"props":637,"children":638},{},[],{"type":40,"tag":55,"props":640,"children":642},{"id":641},"mrcr-nemo-gym-simple_agent-path-branch-here-too",[643,645,651],{"type":46,"value":644},"MRCR (NeMo Gym ",{"type":40,"tag":67,"props":646,"children":648},{"className":647},[],[649],{"type":46,"value":650},"simple_agent",{"type":46,"value":652},") path — branch here too",{"type":40,"tag":49,"props":654,"children":655},{},[656,658,663,665,669,671,676,677,682,684,689,691,696],{"type":46,"value":657},"A 0.2.6 ",{"type":40,"tag":67,"props":659,"children":661},{"className":660},[],[662],{"type":46,"value":400},{"type":46,"value":664}," task like GDPVal and equally ",{"type":40,"tag":125,"props":666,"children":667},{},[668],{"type":46,"value":407},{"type":46,"value":670},", but far simpler:\n",{"type":40,"tag":67,"props":672,"children":674},{"className":673},[],[675],{"type":46,"value":650},{"type":46,"value":303},{"type":40,"tag":125,"props":678,"children":679},{},[680],{"type":46,"value":681},"no SIF, no judge, no Tavily",{"type":46,"value":683}," — deterministic prefix-gated\ngrading, ",{"type":40,"tag":67,"props":685,"children":687},{"className":686},[],[688],{"type":46,"value":548},{"type":46,"value":690}," the only secret. ",{"type":40,"tag":125,"props":692,"children":693},{},[694],{"type":46,"value":695},"Not an AA benchmark",{"type":46,"value":697}," — never generate it\nfor an \"AA\" request. If the user asks for MRCR:",{"type":40,"tag":223,"props":699,"children":700},{},[701,727,790,851,900],{"type":40,"tag":227,"props":702,"children":703},{},[704,705,714,716,725],{"type":46,"value":231},{"type":40,"tag":125,"props":706,"children":707},{},[708],{"type":40,"tag":67,"props":709,"children":711},{"className":710},[],[712],{"type":46,"value":713},"recipes\u002Ftasks\u002Fgym\u002Fmrcr.md",{"type":46,"value":715},"; start from\n",{"type":40,"tag":125,"props":717,"children":718},{},[719],{"type":40,"tag":67,"props":720,"children":722},{"className":721},[],[723],{"type":46,"value":724},"recipes\u002Fexamples\u002Fgym\u002Fexample_mrcr.yaml",{"type":46,"value":726}," (1M variant, like the golden).",{"type":40,"tag":227,"props":728,"children":729},{},[730,735,737,743,745,751,752,758,760,766,768,773,775,781,783,789],{"type":40,"tag":125,"props":731,"children":732},{},[733],{"type":46,"value":734},"Pick the variant first",{"type":46,"value":736}," (",{"type":40,"tag":67,"props":738,"children":740},{"className":739},[],[741],{"type":46,"value":742},"config_n3_1m",{"type":46,"value":744}," \u002F ",{"type":40,"tag":67,"props":746,"children":748},{"className":747},[],[749],{"type":46,"value":750},"config_n3_128k",{"type":46,"value":744},{"type":40,"tag":67,"props":753,"children":755},{"className":754},[],[756],{"type":46,"value":757},"config",{"type":46,"value":759},") — it\nsets the context cap, dataset ",{"type":40,"tag":761,"props":762,"children":763},"em",{},[764],{"type":46,"value":765},"and",{"type":46,"value":767}," metric prefix; the three are not\ncomparable; set it in ",{"type":40,"tag":125,"props":769,"children":770},{},[771],{"type":46,"value":772},"both",{"type":46,"value":774}," ",{"type":40,"tag":67,"props":776,"children":778},{"className":777},[],[779],{"type":46,"value":780},"data_prep_params",{"type":46,"value":782}," and ",{"type":40,"tag":67,"props":784,"children":786},{"className":785},[],[787],{"type":46,"value":788},"collect_rollout_params",{"type":46,"value":258},{"type":40,"tag":227,"props":791,"children":792},{},[793,798,800,805,807,812,814,819,821,827,829,835,837,843,844,849],{"type":40,"tag":67,"props":794,"children":796},{"className":795},[],[797],{"type":46,"value":317},{"type":46,"value":799},": ",{"type":40,"tag":67,"props":801,"children":803},{"className":802},[],[804],{"type":46,"value":548},{"type":46,"value":806}," (dataset + n3 tokenizer are gated) plus\n",{"type":40,"tag":67,"props":808,"children":810},{"className":809},[],[811],{"type":46,"value":584},{"type":46,"value":813}," (the ",{"type":40,"tag":67,"props":815,"children":817},{"className":816},[],[818],{"type":46,"value":592},{"type":46,"value":820}," installs ",{"type":40,"tag":67,"props":822,"children":824},{"className":823},[],[825],{"type":46,"value":826},"tiktoken",{"type":46,"value":828}," +\n",{"type":40,"tag":67,"props":830,"children":832},{"className":831},[],[833],{"type":46,"value":834},"transformers",{"type":46,"value":836},"; prepare fails without it) and\n",{"type":40,"tag":67,"props":838,"children":840},{"className":839},[],[841],{"type":46,"value":842},"NEMO_EVALUATOR_TRUST_UNLISTED_TASKS=1",{"type":46,"value":736},{"type":40,"tag":67,"props":845,"children":847},{"className":846},[],[848],{"type":46,"value":400},{"type":46,"value":850}," is not in the FDF map).",{"type":40,"tag":227,"props":852,"children":853},{},[854,859,861,867,869,875,877,883,885,891,893,898],{"type":40,"tag":125,"props":855,"children":856},{},[857],{"type":46,"value":858},"MRCR needs a git-backed Gym image.",{"type":46,"value":860}," The pin is newer than any image's baked\nGym and must apply, so the template's ",{"type":40,"tag":67,"props":862,"children":864},{"className":863},[],[865],{"type":46,"value":866},"container:",{"type":46,"value":868}," is ",{"type":40,"tag":67,"props":870,"children":872},{"className":871},[],[873],{"type":46,"value":874},"???",{"type":46,"value":876}," and the bootstrap\nexits 1 on a non-git ",{"type":40,"tag":67,"props":878,"children":880},{"className":879},[],[881],{"type":46,"value":882},"\u002Fopt\u002FGym",{"type":46,"value":884}," (the public ",{"type":40,"tag":67,"props":886,"children":888},{"className":887},[],[889],{"type":46,"value":890},"eval-factory\u002Fnemo-gym:*",{"type":46,"value":892}," images).\nNVIDIA-internal: ",{"type":40,"tag":67,"props":894,"children":896},{"className":895},[],[897],{"type":46,"value":293},{"type":46,"value":899}," Step 3d names a working image.",{"type":40,"tag":227,"props":901,"children":902},{},[903,905,911,912,918,919,925,927,932,934,940],{"type":46,"value":904},"Long-context deploy (",{"type":40,"tag":67,"props":906,"children":908},{"className":907},[],[909],{"type":46,"value":910},"--max-model-len 1100000",{"type":46,"value":828},{"type":40,"tag":67,"props":913,"children":915},{"className":914},[],[916],{"type":46,"value":917},"VLLM_ALLOW_LONG_MAX_MODEL_LEN=1",{"type":46,"value":303},{"type":40,"tag":67,"props":920,"children":922},{"className":921},[],[923],{"type":46,"value":924},"gpu_memory_utilization: 0.95",{"type":46,"value":926},",\nmulti-instance fan-out); ",{"type":40,"tag":125,"props":928,"children":929},{},[930],{"type":46,"value":931},"never cap output tokens",{"type":46,"value":933},"; report the needle-count\nstrata alongside ",{"type":40,"tag":67,"props":935,"children":937},{"className":936},[],[938],{"type":46,"value":939},"pass@1\u002Faccuracy",{"type":46,"value":258},{"type":40,"tag":110,"props":942,"children":943},{},[],{"type":40,"tag":55,"props":945,"children":947},{"id":946},"step-1-prerequisites",[948],{"type":46,"value":949},"Step 1 — Prerequisites",{"type":40,"tag":951,"props":952,"children":953},"blockquote",{},[954],{"type":40,"tag":49,"props":955,"children":956},{},[957,962,964,970,972,983,985,991,993,999,1001,1007,1009,1015],{"type":40,"tag":125,"props":958,"children":959},{},[960],{"type":46,"value":961},"SSH:",{"type":46,"value":963}," NEL opens its own connection as a bare ",{"type":40,"tag":67,"props":965,"children":967},{"className":966},[],[968],{"type":46,"value":969},"ssh \u003Cuser>@\u003Chost>",{"type":46,"value":971}," with ",{"type":40,"tag":125,"props":973,"children":974},{},[975,977],{"type":46,"value":976},"no ",{"type":40,"tag":67,"props":978,"children":980},{"className":979},[],[981],{"type":46,"value":982},"-i",{"type":46,"value":984},", so a\nnon-default key filename makes every submit fail with ",{"type":40,"tag":67,"props":986,"children":988},{"className":987},[],[989],{"type":46,"value":990},"Permission denied (publickey,password)",{"type":46,"value":992},"\neven though your own tooling works. Add an ",{"type":40,"tag":67,"props":994,"children":996},{"className":995},[],[997],{"type":46,"value":998},"IdentityFile",{"type":46,"value":1000}," entry to ",{"type":40,"tag":67,"props":1002,"children":1004},{"className":1003},[],[1005],{"type":46,"value":1006},"~\u002F.ssh\u002Fconfig",{"type":46,"value":1008}," per cluster and\nverify with ",{"type":40,"tag":67,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":46,"value":1014},"ssh -o BatchMode=yes \u003Cuser>@\u003Chost> true",{"type":46,"value":258},{"type":40,"tag":951,"props":1017,"children":1018},{},[1019],{"type":40,"tag":49,"props":1020,"children":1021},{},[1022,1027,1029,1035,1037,1043,1045,1051,1053,1059,1061,1067,1069,1074,1076,1081],{"type":40,"tag":125,"props":1023,"children":1024},{},[1025],{"type":46,"value":1026},"Config placement gotchas",{"type":46,"value":1028},", each of which costs one failed submission: ",{"type":40,"tag":67,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":46,"value":1034},"sbatch_comment",{"type":46,"value":1036}," belongs\nunder ",{"type":40,"tag":67,"props":1038,"children":1040},{"className":1039},[],[1041],{"type":46,"value":1042},"execution:",{"type":46,"value":1044},", not ",{"type":40,"tag":67,"props":1046,"children":1048},{"className":1047},[],[1049],{"type":46,"value":1050},"cluster:",{"type":46,"value":1052}," (otherwise inert, and the idle-GPU reaper kills the job); pyxis\nneeds ",{"type":40,"tag":67,"props":1054,"children":1056},{"className":1055},[],[1057],{"type":46,"value":1058},"registry#path:tag",{"type":46,"value":1060}," for non-DockerHub images or it prepends ",{"type":40,"tag":67,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":46,"value":1066},"docker.io",{"type":46,"value":1068}," and 404s; NEL rejects\n",{"type":40,"tag":125,"props":1070,"children":1071},{},[1072],{"type":46,"value":1073},"file",{"type":46,"value":1075}," mounts (\"Mount paths must be directories\") — put file overrides in ",{"type":40,"tag":67,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":46,"value":592},{"type":46,"value":258},{"type":40,"tag":49,"props":1083,"children":1084},{},[1085,1086,1092,1094,1100,1102,1107],{"type":46,"value":284},{"type":40,"tag":67,"props":1087,"children":1089},{"className":1088},[],[1090],{"type":46,"value":1091},"nel --version",{"type":46,"value":1093},"; if missing, instruct ",{"type":40,"tag":67,"props":1095,"children":1097},{"className":1096},[],[1098],{"type":46,"value":1099},"pip install nemo-evaluator-launcher",{"type":46,"value":1101},". If user has an existing config, skip to Step 8 (optionally review for ",{"type":40,"tag":67,"props":1103,"children":1105},{"className":1104},[],[1106],{"type":46,"value":874},{"type":46,"value":1108}," and quantization flags first).",{"type":40,"tag":49,"props":1110,"children":1111},{},[1112,1124,1126,1131,1133,1138,1140,1145,1147,1152,1154,1158,1160,1165,1167,1173,1175,1181,1183,1188,1190,1196,1198,1203,1205,1211,1213,1218,1220,1226,1228,1233,1235,1241,1242,1248,1250,1256],{"type":40,"tag":125,"props":1113,"children":1114},{},[1115,1117,1122],{"type":46,"value":1116},"Set up ",{"type":40,"tag":67,"props":1118,"children":1120},{"className":1119},[],[1121],{"type":46,"value":317},{"type":46,"value":1123}," now (not Step 8).",{"type":46,"value":1125}," The working ",{"type":40,"tag":67,"props":1127,"children":1129},{"className":1128},[],[1130],{"type":46,"value":317},{"type":46,"value":1132}," lives at the ",{"type":40,"tag":125,"props":1134,"children":1135},{},[1136],{"type":46,"value":1137},"workspace root",{"type":46,"value":1139}," — the directory you run ",{"type":40,"tag":67,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":46,"value":277},{"type":46,"value":1146}," from — matching ",{"type":40,"tag":67,"props":1148,"children":1150},{"className":1149},[],[1151],{"type":46,"value":293},{"type":46,"value":1153},"'s convention; do ",{"type":40,"tag":125,"props":1155,"children":1156},{},[1157],{"type":46,"value":136},{"type":46,"value":1159}," create it under the skill dir. (NEL does not discover ",{"type":40,"tag":67,"props":1161,"children":1163},{"className":1162},[],[1164],{"type":46,"value":317},{"type":46,"value":1166}," by path: it reads secrets from the shell env via the ",{"type":40,"tag":67,"props":1168,"children":1170},{"className":1169},[],[1171],{"type":46,"value":1172},"host:",{"type":46,"value":1174}," prefix after you ",{"type":40,"tag":67,"props":1176,"children":1178},{"className":1177},[],[1179],{"type":46,"value":1180},"source",{"type":46,"value":1182},", so the location is purely ",{"type":40,"tag":761,"props":1184,"children":1185},{},[1186],{"type":46,"value":1187},"which file you source",{"type":46,"value":1189}," before ",{"type":40,"tag":67,"props":1191,"children":1193},{"className":1192},[],[1194],{"type":46,"value":1195},"nel run",{"type":46,"value":1197},". Keeping the single ",{"type":40,"tag":67,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":46,"value":317},{"type":46,"value":1204}," at the workspace root avoids a stale duplicate under the symlinked, shared ",{"type":40,"tag":67,"props":1206,"children":1208},{"className":1207},[],[1209],{"type":46,"value":1210},".agents\u002F",{"type":46,"value":1212}," skill tree.) For judge-scored \u002F user-sim tasks (HLE, AA-LCR, Tau2), seed it from the template if absent — the template ships under the skill dir, the working ",{"type":40,"tag":67,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":46,"value":317},{"type":46,"value":1219}," does not: ",{"type":40,"tag":67,"props":1221,"children":1223},{"className":1222},[],[1224],{"type":46,"value":1225},"[ -f .env ] || cp \"$SKILL_DIR\u002Frecipes\u002Fenv.example\" .env",{"type":46,"value":1227},". Then try ",{"type":40,"tag":67,"props":1229,"children":1231},{"className":1230},[],[1232],{"type":46,"value":293},{"type":46,"value":1234}," (if available) to fill the judge ",{"type":40,"tag":67,"props":1236,"children":1238},{"className":1237},[],[1239],{"type":46,"value":1240},"model_id",{"type":46,"value":185},{"type":40,"tag":67,"props":1243,"children":1245},{"className":1244},[],[1246],{"type":46,"value":1247},"url",{"type":46,"value":1249}," rows (user adds the secret key). Needed before Step 5, which substitutes those values into task ",{"type":40,"tag":67,"props":1251,"children":1253},{"className":1252},[],[1254],{"type":46,"value":1255},"\u003CVAR>",{"type":46,"value":1257}," placeholders.",{"type":40,"tag":49,"props":1259,"children":1260},{},[1261,1273,1275,1280,1282,1288,1290,1295,1297,1303,1305,1310,1312,1317],{"type":40,"tag":125,"props":1262,"children":1263},{},[1264,1266,1271],{"type":46,"value":1265},"Secret safety — never open ",{"type":40,"tag":67,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":46,"value":317},{"type":46,"value":1272}," with Read\u002FWrite\u002FEdit.",{"type":46,"value":1274}," The harness mirrors later edits of any agent-opened file into the transcript, so touching ",{"type":40,"tag":67,"props":1276,"children":1278},{"className":1277},[],[1279],{"type":46,"value":317},{"type":46,"value":1281}," leaks the keys the user adds afterward. Use shell only (",{"type":40,"tag":67,"props":1283,"children":1285},{"className":1284},[],[1286],{"type":46,"value":1287},"cp",{"type":46,"value":1289}," to create, ",{"type":40,"tag":67,"props":1291,"children":1293},{"className":1292},[],[1294],{"type":46,"value":1180},{"type":46,"value":1296}," to load — neither echoes); edit ",{"type":40,"tag":67,"props":1298,"children":1300},{"className":1299},[],[1301],{"type":46,"value":1302},"env.example",{"type":46,"value":1304},", never ",{"type":40,"tag":67,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":46,"value":317},{"type":46,"value":1311},"; leave value entry to the user \u002F ",{"type":40,"tag":67,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":46,"value":293},{"type":46,"value":258},{"type":40,"tag":49,"props":1319,"children":1320},{},[1321,1326],{"type":40,"tag":125,"props":1322,"children":1323},{},[1324],{"type":46,"value":1325},"Task recipes",{"type":46,"value":1327}," (always read before editing the relevant task in the config):",{"type":40,"tag":1329,"props":1330,"children":1331},"ul",{},[1332,1351,1376,1437],{"type":40,"tag":227,"props":1333,"children":1334},{},[1335,1337,1343,1345],{"type":46,"value":1336},"AA Index v2 suite (default for quantized-checkpoint validation, see ",{"type":40,"tag":67,"props":1338,"children":1340},{"className":1339},[],[1341],{"type":46,"value":1342},"references\u002Fquantization-benchmarks.md",{"type":46,"value":1344},"): ",{"type":40,"tag":67,"props":1346,"children":1348},{"className":1347},[],[1349],{"type":46,"value":1350},"recipes\u002Ftasks\u002Faa\u002F{gpqa_diamond,hle,lcr,scicode,ifbench,mmmu_pro,tau2_bench_telecom,omniscience}.md",{"type":40,"tag":227,"props":1352,"children":1353},{},[1354,1356,1362,1363,1369,1370],{"type":46,"value":1355},"Optional: ",{"type":40,"tag":67,"props":1357,"children":1359},{"className":1358},[],[1360],{"type":46,"value":1361},"recipes\u002Ftasks\u002Fmmlu_pro.md",{"type":46,"value":303},{"type":40,"tag":67,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":46,"value":1368},"recipes\u002Ftasks\u002Faime_2025.md",{"type":46,"value":303},{"type":40,"tag":67,"props":1371,"children":1373},{"className":1372},[],[1374],{"type":46,"value":1375},"recipes\u002Ftasks\u002Flivecodebench.md",{"type":40,"tag":227,"props":1377,"children":1378},{},[1379,1384,1386,1391,1393,1398,1400,1406,1408,1413,1415,1421,1423,1428,1430,1435],{"type":40,"tag":125,"props":1380,"children":1381},{},[1382],{"type":46,"value":1383},"nel-next only",{"type":46,"value":1385}," (different evaluator — see the nel-next section below, NOT the 0.2.6 steps): shared reference ",{"type":40,"tag":67,"props":1387,"children":1389},{"className":1388},[],[1390],{"type":46,"value":240},{"type":46,"value":1392}," + per-benchmark recipes ",{"type":40,"tag":67,"props":1394,"children":1396},{"className":1395},[],[1397],{"type":46,"value":248},{"type":46,"value":1399}," (agentic). The ",{"type":40,"tag":67,"props":1401,"children":1403},{"className":1402},[],[1404],{"type":46,"value":1405},"aa_next\u002F",{"type":46,"value":1407}," dir holds tasks that require ",{"type":40,"tag":67,"props":1409,"children":1411},{"className":1410},[],[1412],{"type":46,"value":159},{"type":46,"value":1414}," 0.4.x (the package; ",{"type":40,"tag":67,"props":1416,"children":1418},{"className":1417},[],[1419],{"type":46,"value":1420},"nemo-evaluator-next",{"type":46,"value":1422}," is the eval ",{"type":40,"tag":761,"props":1424,"children":1425},{},[1426],{"type":46,"value":1427},"image",{"type":46,"value":1429}," repo); ",{"type":40,"tag":67,"props":1431,"children":1433},{"className":1432},[],[1434],{"type":46,"value":415},{"type":46,"value":1436}," is the 0.2.6 suite.",{"type":40,"tag":227,"props":1438,"children":1439},{},[1440,1445,1447,1453,1455,1461,1463,1469,1471,1476,1478,1483,1484,1488,1490,1494,1496,1516,1518],{"type":40,"tag":125,"props":1441,"children":1442},{},[1443],{"type":46,"value":1444},"NeMo Gym tasks",{"type":46,"value":1446}," — ",{"type":40,"tag":67,"props":1448,"children":1450},{"className":1449},[],[1451],{"type":46,"value":1452},"recipes\u002Ftasks\u002Fgym\u002F*.md",{"type":46,"value":1454},", with self-contained examples at ",{"type":40,"tag":67,"props":1456,"children":1458},{"className":1457},[],[1459],{"type":46,"value":1460},"recipes\u002Fexamples\u002Fgym\u002Fexample_\u003Ctask>.yaml",{"type":46,"value":1462},". The ",{"type":40,"tag":67,"props":1464,"children":1466},{"className":1465},[],[1467],{"type":46,"value":1468},"gym\u002F",{"type":46,"value":1470}," dir groups by ",{"type":40,"tag":125,"props":1472,"children":1473},{},[1474],{"type":46,"value":1475},"harness",{"type":46,"value":1477}," (0.2.6 ",{"type":40,"tag":67,"props":1479,"children":1481},{"className":1480},[],[1482],{"type":46,"value":400},{"type":46,"value":169},{"type":40,"tag":125,"props":1485,"children":1486},{},[1487],{"type":46,"value":136},{"type":46,"value":1489}," by suite membership, so read AA membership per task from the table below — never from the path. Every gym task is ",{"type":40,"tag":125,"props":1491,"children":1492},{},[1493],{"type":46,"value":407},{"type":46,"value":1495},": generated as its own config from its example, one gym eval per config, ",{"type":40,"tag":125,"props":1497,"children":1498},{},[1499,1501,1506,1508,1514],{"type":46,"value":1500},"never merged into the ",{"type":40,"tag":67,"props":1502,"children":1504},{"className":1503},[],[1505],{"type":46,"value":415},{"type":46,"value":1507}," multi-task ",{"type":40,"tag":67,"props":1509,"children":1511},{"className":1510},[],[1512],{"type":46,"value":1513},"tasks",{"type":46,"value":1515}," list",{"type":46,"value":1517}," and never mixed with each other.",{"type":40,"tag":1519,"props":1520,"children":1521},"table",{},[1522,1551],{"type":40,"tag":1523,"props":1524,"children":1525},"thead",{},[1526],{"type":40,"tag":1527,"props":1528,"children":1529},"tr",{},[1530,1536,1541,1546],{"type":40,"tag":1531,"props":1532,"children":1533},"th",{},[1534],{"type":46,"value":1535},"Task",{"type":40,"tag":1531,"props":1537,"children":1538},{},[1539],{"type":46,"value":1540},"Recipe \u002F example",{"type":40,"tag":1531,"props":1542,"children":1543},{},[1544],{"type":46,"value":1545},"In AA suite?",{"type":40,"tag":1531,"props":1547,"children":1548},{},[1549],{"type":46,"value":1550},"Generate when",{"type":40,"tag":1552,"props":1553,"children":1554},"tbody",{},[1555,1603],{"type":40,"tag":1527,"props":1556,"children":1557},{},[1558,1569,1590,1598],{"type":40,"tag":1559,"props":1560,"children":1561},"td",{},[1562,1567],{"type":40,"tag":125,"props":1563,"children":1564},{},[1565],{"type":46,"value":1566},"GDPVal",{"type":46,"value":1568}," (Stirrup agent, agentic)",{"type":40,"tag":1559,"props":1570,"children":1571},{},[1572,1577,1579,1584,1585],{"type":40,"tag":67,"props":1573,"children":1575},{"className":1574},[],[1576],{"type":46,"value":444},{"type":46,"value":1578}," + ",{"type":40,"tag":67,"props":1580,"children":1582},{"className":1581},[],[1583],{"type":46,"value":433},{"type":46,"value":303},{"type":40,"tag":67,"props":1586,"children":1588},{"className":1587},[],[1589],{"type":46,"value":459},{"type":40,"tag":1559,"props":1591,"children":1592},{},[1593],{"type":40,"tag":125,"props":1594,"children":1595},{},[1596],{"type":46,"value":1597},"Yes",{"type":40,"tag":1559,"props":1599,"children":1600},{},[1601],{"type":46,"value":1602},"any AA request (see the AA rule below)",{"type":40,"tag":1527,"props":1604,"children":1605},{},[1606,1616,1630,1638],{"type":40,"tag":1559,"props":1607,"children":1608},{},[1609,1614],{"type":40,"tag":125,"props":1610,"children":1611},{},[1612],{"type":46,"value":1613},"MRCR",{"type":46,"value":1615}," (simple agent, long-context)",{"type":40,"tag":1559,"props":1617,"children":1618},{},[1619,1624,1625],{"type":40,"tag":67,"props":1620,"children":1622},{"className":1621},[],[1623],{"type":46,"value":713},{"type":46,"value":303},{"type":40,"tag":67,"props":1626,"children":1628},{"className":1627},[],[1629],{"type":46,"value":724},{"type":40,"tag":1559,"props":1631,"children":1632},{},[1633],{"type":40,"tag":125,"props":1634,"children":1635},{},[1636],{"type":46,"value":1637},"No",{"type":40,"tag":1559,"props":1639,"children":1640},{},[1641],{"type":46,"value":1642},"only when the user asks for MRCR by name, or for long-context coverage",{"type":40,"tag":49,"props":1644,"children":1645},{},[1646,1651,1653,1659,1661,1666,1667,1672,1674,1679,1680,1685],{"type":40,"tag":125,"props":1647,"children":1648},{},[1649],{"type":46,"value":1650},"AA rule:",{"type":46,"value":1652}," If the user mentions \"AA\" \u002F \"Artificial Analysis\", generate the ",{"type":40,"tag":67,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":46,"value":1658},"recipes\u002Ftasks\u002Faa\u002F",{"type":46,"value":1660}," tasks (one multi-task config) ",{"type":40,"tag":125,"props":1662,"children":1663},{},[1664],{"type":46,"value":1665},"plus a companion standalone GDPVal config",{"type":46,"value":736},{"type":40,"tag":67,"props":1668,"children":1670},{"className":1669},[],[1671],{"type":46,"value":444},{"type":46,"value":1673},", via the GDPVal branch) — GDPVal is part of the AA suite but a different harness, so it's its own config, never added to the ",{"type":40,"tag":67,"props":1675,"children":1677},{"className":1676},[],[1678],{"type":46,"value":415},{"type":46,"value":774},{"type":40,"tag":67,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":46,"value":1513},{"type":46,"value":1686}," list. Do not add MMLU-Pro, AIME 2025, or LiveCodeBench unless explicitly asked. GDPVal is the heaviest AA task (standalone, multi-hour, needs the SIF sandbox + judge) — surface it and let the user opt out per run.",{"type":40,"tag":49,"props":1688,"children":1689},{},[1690,1695],{"type":40,"tag":125,"props":1691,"children":1692},{},[1693],{"type":46,"value":1694},"Shortcut path",{"type":46,"value":1696}," (when task list is known up front, e.g. \"run AA\"):",{"type":40,"tag":223,"props":1698,"children":1699},{},[1700,1705,1718,1730,1953],{"type":40,"tag":227,"props":1701,"children":1702},{},[1703],{"type":46,"value":1704},"Read the task reference file(s).",{"type":40,"tag":227,"props":1706,"children":1707},{},[1708,1710,1716],{"type":46,"value":1709},"Use ",{"type":40,"tag":67,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":46,"value":1715},"recipes\u002Fexamples\u002Fexample_eval.yaml",{"type":46,"value":1717}," as the base.",{"type":40,"tag":227,"props":1719,"children":1720},{},[1721,1723,1728],{"type":46,"value":1722},"Copy the YAML fragment(s) into ",{"type":40,"tag":67,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":46,"value":219},{"type":46,"value":1729},", applying any per-task notes.",{"type":40,"tag":227,"props":1731,"children":1732},{},[1733,1738,1740,1745,1747,1753,1755,1760,1761,1767,1769,1773,1775,1781,1783,1788,1790,1795,1797,1803,1804,1810,1811,1817,1819,1825,1827,1831,1833,1839,1840,1846,1848,1854,1855,1861,1862,1868,1870,1876,1877,1883,1884,1890,1892,1897,1899,1905,1907,1913,1915,1920,1922,1927,1929,1935,1937,1943,1945,1951],{"type":40,"tag":125,"props":1734,"children":1735},{},[1736],{"type":46,"value":1737},"MLflow auto-export is on by default",{"type":46,"value":1739}," — it needs ",{"type":40,"tag":125,"props":1741,"children":1742},{},[1743],{"type":46,"value":1744},"two",{"type":46,"value":1746}," pieces, both in ",{"type":40,"tag":67,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":46,"value":1752},"example_eval.yaml",{"type":46,"value":1754},": (a) the ",{"type":40,"tag":125,"props":1756,"children":1757},{},[1758],{"type":46,"value":1759},"trigger",{"type":46,"value":774},{"type":40,"tag":67,"props":1762,"children":1764},{"className":1763},[],[1765],{"type":46,"value":1766},"execution.auto_export.destinations: [mlflow]",{"type":46,"value":1768}," (without it the run is ",{"type":40,"tag":761,"props":1770,"children":1771},{},[1772],{"type":46,"value":136},{"type":46,"value":1774}," uploaded), and (b) the ",{"type":40,"tag":67,"props":1776,"children":1778},{"className":1777},[],[1779],{"type":46,"value":1780},"export.mlflow",{"type":46,"value":1782}," block that configures it. In the ",{"type":40,"tag":67,"props":1784,"children":1786},{"className":1785},[],[1787],{"type":46,"value":1780},{"type":46,"value":1789}," block use ",{"type":40,"tag":125,"props":1791,"children":1792},{},[1793],{"type":46,"value":1794},"literal",{"type":46,"value":1796}," values for ",{"type":40,"tag":67,"props":1798,"children":1800},{"className":1799},[],[1801],{"type":46,"value":1802},"experiment_name",{"type":46,"value":744},{"type":40,"tag":67,"props":1805,"children":1807},{"className":1806},[],[1808],{"type":46,"value":1809},"description",{"type":46,"value":744},{"type":40,"tag":67,"props":1812,"children":1814},{"className":1813},[],[1815],{"type":46,"value":1816},"tags",{"type":46,"value":1818}," — substitute the actual ",{"type":40,"tag":67,"props":1820,"children":1822},{"className":1821},[],[1823],{"type":46,"value":1824},"served_model_name",{"type":46,"value":1826}," and sampling params. Do ",{"type":40,"tag":125,"props":1828,"children":1829},{},[1830],{"type":46,"value":136},{"type":46,"value":1832}," use ",{"type":40,"tag":67,"props":1834,"children":1836},{"className":1835},[],[1837],{"type":46,"value":1838},"${deployment.*}",{"type":46,"value":744},{"type":40,"tag":67,"props":1841,"children":1843},{"className":1842},[],[1844],{"type":46,"value":1845},"${evaluation.*}",{"type":46,"value":1847}," cross-references: with auto-export on, NEL resolves the export block at submit time in a scope without those nodes and fails with ",{"type":40,"tag":67,"props":1849,"children":1851},{"className":1850},[],[1852],{"type":46,"value":1853},"Interpolation key '...' not found",{"type":46,"value":736},{"type":40,"tag":67,"props":1856,"children":1858},{"className":1857},[],[1859],{"type":46,"value":1860},"${oc.env:USER}",{"type":46,"value":782},{"type":40,"tag":67,"props":1863,"children":1865},{"className":1864},[],[1866],{"type":46,"value":1867},"${oc.env:MLFLOW_TRACKING_URI}",{"type":46,"value":1869}," are fine — they're env vars). Because these literals can't interpolate, keep the ",{"type":40,"tag":67,"props":1871,"children":1873},{"className":1872},[],[1874],{"type":46,"value":1875},"temperature",{"type":46,"value":744},{"type":40,"tag":67,"props":1878,"children":1880},{"className":1879},[],[1881],{"type":46,"value":1882},"top_p",{"type":46,"value":744},{"type":40,"tag":67,"props":1885,"children":1887},{"className":1886},[],[1888],{"type":46,"value":1889},"max_new_tokens",{"type":46,"value":1891}," tags ",{"type":40,"tag":125,"props":1893,"children":1894},{},[1895],{"type":46,"value":1896},"equal to",{"type":46,"value":1898}," the top-level ",{"type":40,"tag":67,"props":1900,"children":1902},{"className":1901},[],[1903],{"type":46,"value":1904},"params",{"type":46,"value":1906}," and update both in the same edit — they're the only queryable record of sampling in MLflow (NEL doesn't log them as run params), so a stale tag silently misreports the run. ",{"type":40,"tag":67,"props":1908,"children":1910},{"className":1909},[],[1911],{"type":46,"value":1912},"tracking_uri",{"type":46,"value":1914}," = ",{"type":40,"tag":67,"props":1916,"children":1918},{"className":1917},[],[1919],{"type":46,"value":1867},{"type":46,"value":1921}," from ",{"type":40,"tag":67,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":46,"value":293},{"type":46,"value":1928}," (not hand-filled), and auto-export needs ",{"type":40,"tag":67,"props":1930,"children":1932},{"className":1931},[],[1933],{"type":46,"value":1934},"execution.cpu_partition",{"type":46,"value":1936}," (e.g. gcp-nrt ",{"type":40,"tag":67,"props":1938,"children":1940},{"className":1939},[],[1941],{"type":46,"value":1942},"cpu",{"type":46,"value":1944},") — it's a separate CPU-only sbatch that GPU-only partitions reject (",{"type":40,"tag":67,"props":1946,"children":1948},{"className":1947},[],[1949],{"type":46,"value":1950},"Cannot find GPU specification",{"type":46,"value":1952},"), silently dropping the link.",{"type":40,"tag":227,"props":1954,"children":1955},{},[1956],{"type":46,"value":1957},"Proceed to Step 3, then Step 4, then Step 7.5\u002F8. Skip Step 2's 5-question flow.",{"type":40,"tag":110,"props":1959,"children":1960},{},[],{"type":40,"tag":55,"props":1962,"children":1964},{"id":1963},"step-2-build-base-config-when-not-using-shortcut",[1965],{"type":46,"value":1966},"Step 2 — Build base config (when not using shortcut)",{"type":40,"tag":49,"props":1968,"children":1969},{},[1970,1972,1978,1979,1984],{"type":46,"value":1971},"Ask the 5 questions via AskUserQuestion (categories must match ",{"type":40,"tag":67,"props":1973,"children":1975},{"className":1974},[],[1976],{"type":46,"value":1977},"nel skills build-config --help",{"type":46,"value":1446},{"type":40,"tag":125,"props":1980,"children":1981},{},[1982],{"type":46,"value":1983},"run that first",{"type":46,"value":1985}," to confirm the current option names; CLI options override this list).",{"type":40,"tag":223,"props":1987,"children":1988},{},[1989,1999,2009,2019,2029],{"type":40,"tag":227,"props":1990,"children":1991},{},[1992,1997],{"type":40,"tag":125,"props":1993,"children":1994},{},[1995],{"type":46,"value":1996},"Execution:",{"type":46,"value":1998}," Local \u002F SLURM",{"type":40,"tag":227,"props":2000,"children":2001},{},[2002,2007],{"type":40,"tag":125,"props":2003,"children":2004},{},[2005],{"type":46,"value":2006},"Deployment:",{"type":46,"value":2008}," None (External) \u002F vLLM \u002F SGLang \u002F NIM \u002F TRT-LLM. Prefer vLLM unless the user\u002Fcard says otherwise.",{"type":40,"tag":227,"props":2010,"children":2011},{},[2012,2017],{"type":40,"tag":125,"props":2013,"children":2014},{},[2015],{"type":46,"value":2016},"Auto-export:",{"type":46,"value":2018}," None \u002F MLflow \u002F wandb",{"type":40,"tag":227,"props":2020,"children":2021},{},[2022,2027],{"type":40,"tag":125,"props":2023,"children":2024},{},[2025],{"type":46,"value":2026},"Model type:",{"type":46,"value":2028}," Base \u002F Chat \u002F Reasoning",{"type":40,"tag":227,"props":2030,"children":2031},{},[2032,2037],{"type":40,"tag":125,"props":2033,"children":2034},{},[2035],{"type":46,"value":2036},"Benchmarks",{"type":46,"value":2038}," (multi-select): standard \u002F code \u002F math_reasoning \u002F safety \u002F multilingual",{"type":40,"tag":49,"props":2040,"children":2041},{},[2042],{"type":46,"value":2043},"Build the base:",{"type":40,"tag":98,"props":2045,"children":2049},{"className":2046,"code":2047,"language":2048,"meta":103,"style":103},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","nel skills build-config --execution \u003C...> --deployment \u003C...> --model_type \u003C...> --benchmarks \u003C...> [--export \u003C...>] [--output \u003C...>]\n","bash",[2050],{"type":40,"tag":67,"props":2051,"children":2052},{"__ignoreMap":103},[2053],{"type":40,"tag":2054,"props":2055,"children":2058},"span",{"class":2056,"line":2057},"line",1,[2059,2064,2070,2075,2080,2086,2091,2096,2101,2106,2110,2114,2118,2122,2127,2131,2135,2139,2143,2148,2152,2156,2160,2164,2169,2174,2178,2182,2186,2191,2196,2200,2204,2208,2212],{"type":40,"tag":2054,"props":2060,"children":2062},{"style":2061},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[2063],{"type":46,"value":277},{"type":40,"tag":2054,"props":2065,"children":2067},{"style":2066},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[2068],{"type":46,"value":2069}," skills",{"type":40,"tag":2054,"props":2071,"children":2072},{"style":2066},[2073],{"type":46,"value":2074}," build-config",{"type":40,"tag":2054,"props":2076,"children":2077},{"style":2066},[2078],{"type":46,"value":2079}," --execution",{"type":40,"tag":2054,"props":2081,"children":2083},{"style":2082},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[2084],{"type":46,"value":2085}," \u003C",{"type":40,"tag":2054,"props":2087,"children":2088},{"style":2066},[2089],{"type":46,"value":2090},"..",{"type":40,"tag":2054,"props":2092,"children":2094},{"style":2093},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[2095],{"type":46,"value":258},{"type":40,"tag":2054,"props":2097,"children":2098},{"style":2082},[2099],{"type":46,"value":2100},">",{"type":40,"tag":2054,"props":2102,"children":2103},{"style":2066},[2104],{"type":46,"value":2105}," --deployment",{"type":40,"tag":2054,"props":2107,"children":2108},{"style":2082},[2109],{"type":46,"value":2085},{"type":40,"tag":2054,"props":2111,"children":2112},{"style":2066},[2113],{"type":46,"value":2090},{"type":40,"tag":2054,"props":2115,"children":2116},{"style":2093},[2117],{"type":46,"value":258},{"type":40,"tag":2054,"props":2119,"children":2120},{"style":2082},[2121],{"type":46,"value":2100},{"type":40,"tag":2054,"props":2123,"children":2124},{"style":2066},[2125],{"type":46,"value":2126}," --model_type",{"type":40,"tag":2054,"props":2128,"children":2129},{"style":2082},[2130],{"type":46,"value":2085},{"type":40,"tag":2054,"props":2132,"children":2133},{"style":2066},[2134],{"type":46,"value":2090},{"type":40,"tag":2054,"props":2136,"children":2137},{"style":2093},[2138],{"type":46,"value":258},{"type":40,"tag":2054,"props":2140,"children":2141},{"style":2082},[2142],{"type":46,"value":2100},{"type":40,"tag":2054,"props":2144,"children":2145},{"style":2066},[2146],{"type":46,"value":2147}," --benchmarks",{"type":40,"tag":2054,"props":2149,"children":2150},{"style":2082},[2151],{"type":46,"value":2085},{"type":40,"tag":2054,"props":2153,"children":2154},{"style":2066},[2155],{"type":46,"value":2090},{"type":40,"tag":2054,"props":2157,"children":2158},{"style":2093},[2159],{"type":46,"value":258},{"type":40,"tag":2054,"props":2161,"children":2162},{"style":2082},[2163],{"type":46,"value":2100},{"type":40,"tag":2054,"props":2165,"children":2166},{"style":2093},[2167],{"type":46,"value":2168}," [--export ",{"type":40,"tag":2054,"props":2170,"children":2171},{"style":2082},[2172],{"type":46,"value":2173},"\u003C",{"type":40,"tag":2054,"props":2175,"children":2176},{"style":2066},[2177],{"type":46,"value":2090},{"type":40,"tag":2054,"props":2179,"children":2180},{"style":2093},[2181],{"type":46,"value":258},{"type":40,"tag":2054,"props":2183,"children":2184},{"style":2082},[2185],{"type":46,"value":2100},{"type":40,"tag":2054,"props":2187,"children":2188},{"style":2066},[2189],{"type":46,"value":2190},"]",{"type":40,"tag":2054,"props":2192,"children":2193},{"style":2093},[2194],{"type":46,"value":2195}," [--output ",{"type":40,"tag":2054,"props":2197,"children":2198},{"style":2082},[2199],{"type":46,"value":2173},{"type":40,"tag":2054,"props":2201,"children":2202},{"style":2066},[2203],{"type":46,"value":2090},{"type":40,"tag":2054,"props":2205,"children":2206},{"style":2093},[2207],{"type":46,"value":258},{"type":40,"tag":2054,"props":2209,"children":2210},{"style":2082},[2211],{"type":46,"value":2100},{"type":40,"tag":2054,"props":2213,"children":2214},{"style":2066},[2215],{"type":46,"value":2216},"]\n",{"type":40,"tag":49,"props":2218,"children":2219},{},[2220,2222,2228,2230,2236],{"type":46,"value":2221},"(",{"type":40,"tag":67,"props":2223,"children":2225},{"className":2224},[],[2226],{"type":46,"value":2227},"--output",{"type":46,"value":2229}," omitted = cwd auto-named; directory = dir + auto-name; ",{"type":40,"tag":67,"props":2231,"children":2233},{"className":2232},[],[2234],{"type":46,"value":2235},"*.yaml",{"type":46,"value":2237}," = exact path. Never overwrites.)",{"type":40,"tag":110,"props":2239,"children":2240},{},[],{"type":40,"tag":55,"props":2242,"children":2244},{"id":2243},"step-3-configure-deployment",[2245],{"type":46,"value":2246},"Step 3 — Configure deployment",{"type":40,"tag":49,"props":2248,"children":2249},{},[2250,2255,2257,2262,2263,2269,2270,2276,2277,2283,2285,2291,2293,2299,2301,2306,2308,2314,2315,2321],{"type":40,"tag":125,"props":2251,"children":2252},{},[2253],{"type":46,"value":2254},"Model path.",{"type":46,"value":2256}," Checkpoint path (",{"type":40,"tag":67,"props":2258,"children":2260},{"className":2259},[],[2261],{"type":46,"value":185},{"type":46,"value":303},{"type":40,"tag":67,"props":2264,"children":2266},{"className":2265},[],[2267],{"type":46,"value":2268},".\u002F",{"type":46,"value":303},{"type":40,"tag":67,"props":2271,"children":2273},{"className":2272},[],[2274],{"type":46,"value":2275},"..\u002F",{"type":46,"value":303},{"type":40,"tag":67,"props":2278,"children":2280},{"className":2279},[],[2281],{"type":46,"value":2282},"~",{"type":46,"value":2284},", or exists on disk) → set ",{"type":40,"tag":67,"props":2286,"children":2288},{"className":2287},[],[2289],{"type":46,"value":2290},"deployment.checkpoint_path",{"type":46,"value":2292},", leave ",{"type":40,"tag":67,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":46,"value":2298},"hf_model_handle: null",{"type":46,"value":2300},". Else HF handle (one ",{"type":40,"tag":67,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":46,"value":185},{"type":46,"value":2307},", not on disk) → set ",{"type":40,"tag":67,"props":2309,"children":2311},{"className":2310},[],[2312],{"type":46,"value":2313},"deployment.hf_model_handle",{"type":46,"value":2292},{"type":40,"tag":67,"props":2316,"children":2318},{"className":2317},[],[2319],{"type":46,"value":2320},"checkpoint_path: null",{"type":46,"value":258},{"type":40,"tag":951,"props":2323,"children":2324},{},[2325,2398,2518,2576],{"type":40,"tag":49,"props":2326,"children":2327},{},[2328,2348,2350,2356,2358,2364,2366,2372,2374,2380,2382,2388,2390,2396],{"type":40,"tag":125,"props":2329,"children":2330},{},[2331,2333,2339,2341,2346],{"type":46,"value":2332},"NEVER point ",{"type":40,"tag":67,"props":2334,"children":2336},{"className":2335},[],[2337],{"type":46,"value":2338},"checkpoint_path",{"type":46,"value":2340}," at a HuggingFace ",{"type":40,"tag":761,"props":2342,"children":2343},{},[2344],{"type":46,"value":2345},"cache snapshot",{"type":46,"value":2347}," dir.",{"type":46,"value":2349}," Entries under\n",{"type":40,"tag":67,"props":2351,"children":2353},{"className":2352},[],[2354],{"type":46,"value":2355},"snapshots\u002F\u003Csha>\u002F",{"type":46,"value":2357}," are relative symlinks into ",{"type":40,"tag":67,"props":2359,"children":2361},{"className":2360},[],[2362],{"type":46,"value":2363},"..\u002F..\u002Fblobs\u002F",{"type":46,"value":2365},". NEL mounts only the snapshot dir at\n",{"type":40,"tag":67,"props":2367,"children":2369},{"className":2368},[],[2370],{"type":46,"value":2371},"\u002Fcheckpoint",{"type":46,"value":2373},", so every link dangles in-container and vLLM dies with\n",{"type":40,"tag":67,"props":2375,"children":2377},{"className":2376},[],[2378],{"type":46,"value":2379},"Invalid repository ID or local directory specified: '\u002Fcheckpoint'",{"type":46,"value":2381},". Pre-staging into ",{"type":40,"tag":67,"props":2383,"children":2385},{"className":2384},[],[2386],{"type":46,"value":2387},"HF_HOME",{"type":46,"value":2389},"\ndoes not help — that works for ",{"type":40,"tag":67,"props":2391,"children":2393},{"className":2392},[],[2394],{"type":46,"value":2395},"from_pretrained",{"type":46,"value":2397},", not a mounted directory. Build a hardlink farm\n(same filesystem, no extra space) and point at that:",{"type":40,"tag":98,"props":2399,"children":2401},{"className":2046,"code":2400,"language":2048,"meta":103,"style":103},"find \"$SNAP\" ! -type d -exec sh -c '\n  for f; do\n    rel=${f#\"$SNAP\"\u002F}; mkdir -p \"$DEST\u002F$(dirname \"$rel\")\"\n    ln -f \"$(readlink -f \"$f\")\" \"$DEST\u002F$rel\"\n  done' _ {} +\n",[2402],{"type":40,"tag":67,"props":2403,"children":2404},{"__ignoreMap":103},[2405,2463,2472,2481,2490],{"type":40,"tag":2054,"props":2406,"children":2407},{"class":2056,"line":2057},[2408,2413,2418,2423,2428,2433,2438,2443,2448,2453,2458],{"type":40,"tag":2054,"props":2409,"children":2410},{"style":2061},[2411],{"type":46,"value":2412},"find",{"type":40,"tag":2054,"props":2414,"children":2415},{"style":2082},[2416],{"type":46,"value":2417}," \"",{"type":40,"tag":2054,"props":2419,"children":2420},{"style":2093},[2421],{"type":46,"value":2422},"$SNAP",{"type":40,"tag":2054,"props":2424,"children":2425},{"style":2082},[2426],{"type":46,"value":2427},"\"",{"type":40,"tag":2054,"props":2429,"children":2430},{"style":2066},[2431],{"type":46,"value":2432}," !",{"type":40,"tag":2054,"props":2434,"children":2435},{"style":2066},[2436],{"type":46,"value":2437}," -type",{"type":40,"tag":2054,"props":2439,"children":2440},{"style":2066},[2441],{"type":46,"value":2442}," d",{"type":40,"tag":2054,"props":2444,"children":2445},{"style":2066},[2446],{"type":46,"value":2447}," -exec",{"type":40,"tag":2054,"props":2449,"children":2450},{"style":2066},[2451],{"type":46,"value":2452}," sh",{"type":40,"tag":2054,"props":2454,"children":2455},{"style":2066},[2456],{"type":46,"value":2457}," -c",{"type":40,"tag":2054,"props":2459,"children":2460},{"style":2082},[2461],{"type":46,"value":2462}," '\n",{"type":40,"tag":2054,"props":2464,"children":2466},{"class":2056,"line":2465},2,[2467],{"type":40,"tag":2054,"props":2468,"children":2469},{"style":2066},[2470],{"type":46,"value":2471},"  for f; do\n",{"type":40,"tag":2054,"props":2473,"children":2475},{"class":2056,"line":2474},3,[2476],{"type":40,"tag":2054,"props":2477,"children":2478},{"style":2066},[2479],{"type":46,"value":2480},"    rel=${f#\"$SNAP\"\u002F}; mkdir -p \"$DEST\u002F$(dirname \"$rel\")\"\n",{"type":40,"tag":2054,"props":2482,"children":2484},{"class":2056,"line":2483},4,[2485],{"type":40,"tag":2054,"props":2486,"children":2487},{"style":2066},[2488],{"type":46,"value":2489},"    ln -f \"$(readlink -f \"$f\")\" \"$DEST\u002F$rel\"\n",{"type":40,"tag":2054,"props":2491,"children":2493},{"class":2056,"line":2492},5,[2494,2499,2504,2509,2514],{"type":40,"tag":2054,"props":2495,"children":2496},{"style":2066},[2497],{"type":46,"value":2498},"  done",{"type":40,"tag":2054,"props":2500,"children":2501},{"style":2082},[2502],{"type":46,"value":2503},"'",{"type":40,"tag":2054,"props":2505,"children":2506},{"style":2066},[2507],{"type":46,"value":2508}," _",{"type":40,"tag":2054,"props":2510,"children":2511},{"style":2066},[2512],{"type":46,"value":2513}," {}",{"type":40,"tag":2054,"props":2515,"children":2516},{"style":2066},[2517],{"type":46,"value":828},{"type":40,"tag":49,"props":2519,"children":2520},{},[2521,2522,2528,2529,2535,2537,2542,2544,2550,2552,2558,2560,2566,2568,2574],{"type":46,"value":2221},{"type":40,"tag":67,"props":2523,"children":2525},{"className":2524},[],[2526],{"type":46,"value":2527},"SNAP",{"type":46,"value":185},{"type":40,"tag":67,"props":2530,"children":2532},{"className":2531},[],[2533],{"type":46,"value":2534},"DEST",{"type":46,"value":2536}," must be ",{"type":40,"tag":125,"props":2538,"children":2539},{},[2540],{"type":46,"value":2541},"exported",{"type":46,"value":2543}," — they are read inside a new shell — and the loop preserves\nnested paths; ",{"type":40,"tag":67,"props":2545,"children":2547},{"className":2546},[],[2548],{"type":46,"value":2549},"basename",{"type":46,"value":2551}," would flatten subdirectories into one level.)\nThe same applies to ModelOpt exports whose ",{"type":40,"tag":67,"props":2553,"children":2555},{"className":2554},[],[2556],{"type":46,"value":2557},"--source_ckpt",{"type":46,"value":2559}," was a snapshot: the exporter preserves\nsymlinks, shipping a dangling ",{"type":40,"tag":67,"props":2561,"children":2563},{"className":2562},[],[2564],{"type":46,"value":2565},"tokenizer.json",{"type":46,"value":2567},". Check with ",{"type":40,"tag":67,"props":2569,"children":2571},{"className":2570},[],[2572],{"type":46,"value":2573},"find \"$OUT\" -type l",{"type":46,"value":2575}," before serving.",{"type":40,"tag":49,"props":2577,"children":2578},{},[2579,2599,2600,2605,2607,2612,2614,2620,2622,2628,2630,2635,2637,2642],{"type":40,"tag":125,"props":2580,"children":2581},{},[2582,2584,2589,2591,2597],{"type":46,"value":2583},"Prefer ",{"type":40,"tag":67,"props":2585,"children":2587},{"className":2586},[],[2588],{"type":46,"value":2338},{"type":46,"value":2590}," over ",{"type":40,"tag":67,"props":2592,"children":2594},{"className":2593},[],[2595],{"type":46,"value":2596},"hf_model_handle",{"type":46,"value":2598}," on SLURM",{"type":46,"value":1446},{"type":40,"tag":67,"props":2601,"children":2603},{"className":2602},[],[2604],{"type":46,"value":2596},{"type":46,"value":2606}," isn't reliably mounted at ",{"type":40,"tag":67,"props":2608,"children":2610},{"className":2609},[],[2611],{"type":46,"value":2371},{"type":46,"value":2613},", so the deploy dies with ",{"type":40,"tag":67,"props":2615,"children":2617},{"className":2616},[],[2618],{"type":46,"value":2619},"HFValidationError",{"type":46,"value":2621},". To eval an un-staged HF model, stage it first (",{"type":40,"tag":67,"props":2623,"children":2625},{"className":2624},[],[2626],{"type":46,"value":2627},"huggingface_hub.snapshot_download",{"type":46,"value":2629},") and point ",{"type":40,"tag":67,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":46,"value":2338},{"type":46,"value":2636}," at it. See ",{"type":40,"tag":67,"props":2638,"children":2640},{"className":2639},[],[2641],{"type":46,"value":1752},{"type":46,"value":2643}," for why.",{"type":40,"tag":49,"props":2645,"children":2646},{},[2647,2652,2654,2660,2662,2668,2670,2676],{"type":40,"tag":125,"props":2648,"children":2649},{},[2650],{"type":46,"value":2651},"Auto-detect ModelOpt quantization",{"type":46,"value":2653}," (checkpoint paths). Check ",{"type":40,"tag":67,"props":2655,"children":2657},{"className":2656},[],[2658],{"type":46,"value":2659},"config.json",{"type":46,"value":2661}," for ",{"type":40,"tag":67,"props":2663,"children":2665},{"className":2664},[],[2666],{"type":46,"value":2667},"quantization_config",{"type":46,"value":2669}," (or legacy ",{"type":40,"tag":67,"props":2671,"children":2673},{"className":2672},[],[2674],{"type":46,"value":2675},"hf_quant_config.json",{"type":46,"value":2677},"):",{"type":40,"tag":1329,"props":2679,"children":2680},{},[2681,2712],{"type":40,"tag":227,"props":2682,"children":2683},{},[2684,2689,2691,2697,2699,2704,2705,2710],{"type":40,"tag":125,"props":2685,"children":2686},{},[2687],{"type":46,"value":2688},"vLLM:",{"type":46,"value":2690}," no ",{"type":40,"tag":67,"props":2692,"children":2694},{"className":2693},[],[2695],{"type":46,"value":2696},"--quantization",{"type":46,"value":2698}," flag by default — vLLM auto-detects from ",{"type":40,"tag":67,"props":2700,"children":2702},{"className":2701},[],[2703],{"type":46,"value":2667},{"type":46,"value":744},{"type":40,"tag":67,"props":2706,"children":2708},{"className":2707},[],[2709],{"type":46,"value":2675},{"type":46,"value":2711},". Add only when the card, vLLM version, or dry-run error requires it.",{"type":40,"tag":227,"props":2713,"children":2714},{},[2715,2720,2722,2728,2729,2735,2736,2742],{"type":40,"tag":125,"props":2716,"children":2717},{},[2718],{"type":46,"value":2719},"SGLang:",{"type":46,"value":2721}," may need ",{"type":40,"tag":67,"props":2723,"children":2725},{"className":2724},[],[2726],{"type":46,"value":2727},"--quantization modelopt_fp8",{"type":46,"value":744},{"type":40,"tag":67,"props":2730,"children":2732},{"className":2731},[],[2733],{"type":46,"value":2734},"modelopt_fp4",{"type":46,"value":744},{"type":40,"tag":67,"props":2737,"children":2739},{"className":2738},[],[2740],{"type":46,"value":2741},"modelopt",{"type":46,"value":2743}," — verify against installed version.",{"type":40,"tag":49,"props":2745,"children":2746},{},[2747,2749,2755,2757,2763,2764,2770,2772,2778,2779,2783,2784,2790,2792,2798,2800,2806,2808,2813],{"type":46,"value":2748},"Some models need extra vLLM backend env vars (model-card research) — e.g. ",{"type":40,"tag":67,"props":2750,"children":2752},{"className":2751},[],[2753],{"type":46,"value":2754},"VLLM_NVFP4_GEMM_BACKEND=marlin",{"type":46,"value":2756}," (Nemotron Super), or ",{"type":40,"tag":67,"props":2758,"children":2760},{"className":2759},[],[2761],{"type":46,"value":2762},"VLLM_USE_FLASHINFER_MOE_FP4=1",{"type":46,"value":1578},{"type":40,"tag":67,"props":2765,"children":2767},{"className":2766},[],[2768],{"type":46,"value":2769},"VLLM_FLASHINFER_MOE_BACKEND=throughput",{"type":46,"value":2771}," (NVFP4 MoE, e.g. NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4). Put them in ",{"type":40,"tag":67,"props":2773,"children":2775},{"className":2774},[],[2776],{"type":46,"value":2777},"deployment.env_vars",{"type":46,"value":736},{"type":40,"tag":125,"props":2780,"children":2781},{},[2782],{"type":46,"value":136},{"type":46,"value":774},{"type":40,"tag":67,"props":2785,"children":2787},{"className":2786},[],[2788],{"type":46,"value":2789},"command",{"type":46,"value":2791},") with the ",{"type":40,"tag":67,"props":2793,"children":2795},{"className":2794},[],[2796],{"type":46,"value":2797},"lit:",{"type":46,"value":2799}," prefix (",{"type":40,"tag":67,"props":2801,"children":2803},{"className":2802},[],[2804],{"type":46,"value":2805},"VLLM_USE_FLASHINFER_MOE_FP4: lit:1",{"type":46,"value":2807},"); see ",{"type":40,"tag":67,"props":2809,"children":2811},{"className":2810},[],[2812],{"type":46,"value":1752},{"type":46,"value":2814}," and Step 5's prefix rule.",{"type":40,"tag":49,"props":2816,"children":2817},{},[2818],{"type":40,"tag":125,"props":2819,"children":2820},{},[2821,2823,2828],{"type":46,"value":2822},"Auto-detect from ",{"type":40,"tag":67,"props":2824,"children":2826},{"className":2825},[],[2827],{"type":46,"value":2659},{"type":46,"value":2829},":",{"type":40,"tag":1519,"props":2831,"children":2832},{},[2833,2849],{"type":40,"tag":1523,"props":2834,"children":2835},{},[2836],{"type":40,"tag":1527,"props":2837,"children":2838},{},[2839,2844],{"type":40,"tag":1531,"props":2840,"children":2841},{},[2842],{"type":46,"value":2843},"Field",{"type":40,"tag":1531,"props":2845,"children":2846},{},[2847],{"type":46,"value":2848},"Flag",{"type":40,"tag":1552,"props":2850,"children":2851},{},[2852,2873],{"type":40,"tag":1527,"props":2853,"children":2854},{},[2855,2864],{"type":40,"tag":1559,"props":2856,"children":2857},{},[2858],{"type":40,"tag":67,"props":2859,"children":2861},{"className":2860},[],[2862],{"type":46,"value":2863},"max_position_embeddings",{"type":40,"tag":1559,"props":2865,"children":2866},{},[2867],{"type":40,"tag":67,"props":2868,"children":2870},{"className":2869},[],[2871],{"type":46,"value":2872},"--max-model-len \u003Cvalue>",{"type":40,"tag":1527,"props":2874,"children":2875},{},[2876,2887],{"type":40,"tag":1559,"props":2877,"children":2878},{},[2879,2885],{"type":40,"tag":67,"props":2880,"children":2882},{"className":2881},[],[2883],{"type":46,"value":2884},"auto_map",{"type":46,"value":2886}," exists",{"type":40,"tag":1559,"props":2888,"children":2889},{},[2890],{"type":40,"tag":67,"props":2891,"children":2893},{"className":2892},[],[2894],{"type":46,"value":2895},"--trust-remote-code",{"type":40,"tag":2897,"props":2898,"children":2900},"h4",{"id":2899},"cross-check-both-sources-for-vllm-mandatory-neither-replaces-the-other",[2901],{"type":46,"value":2902},"Cross-check both sources for vLLM (mandatory, neither replaces the other)",{"type":40,"tag":49,"props":2904,"children":2905},{},[2906,2917,2919,2925,2926,2932,2933,2939,2941,2946,2948,2954],{"type":40,"tag":125,"props":2907,"children":2908},{},[2909,2911],{"type":46,"value":2910},"Source 1 — ",{"type":40,"tag":67,"props":2912,"children":2914},{"className":2913},[],[2915],{"type":46,"value":2916},"recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>",{"type":46,"value":2918}," (curated vLLM recipes; authoritative for parallelism, family-specific flags like ",{"type":40,"tag":67,"props":2920,"children":2922},{"className":2921},[],[2923],{"type":46,"value":2924},"--reasoning-parser",{"type":46,"value":744},{"type":40,"tag":67,"props":2927,"children":2929},{"className":2928},[],[2930],{"type":46,"value":2931},"--tool-call-parser",{"type":46,"value":744},{"type":40,"tag":67,"props":2934,"children":2936},{"className":2935},[],[2937],{"type":46,"value":2938},"--mm-encoder-tp-mode",{"type":46,"value":2940},", vLLM version, spec-decoding, GPU count). ",{"type":40,"tag":125,"props":2942,"children":2943},{},[2944],{"type":46,"value":2945},"Fetch the page for the EXACT model id, not a base\u002Fsibling",{"type":46,"value":2947}," — variant minimums differ (e.g. MiniMax-M2 ≥0.11.0 vs M2.7 ≥0.20.0). Pin variants via query params (e.g. ",{"type":40,"tag":67,"props":2949,"children":2951},{"className":2950},[],[2952],{"type":46,"value":2953},"?variant=fp8&strategy=single_node_tep",{"type":46,"value":90},{"type":40,"tag":951,"props":2956,"children":2957},{},[2958,2966,3023],{"type":40,"tag":49,"props":2959,"children":2960},{},[2961],{"type":40,"tag":125,"props":2962,"children":2963},{},[2964],{"type":46,"value":2965},"WebFetch caveat — triage the summary:",{"type":40,"tag":223,"props":2967,"children":2968},{},[2969,2987,3013],{"type":40,"tag":227,"props":2970,"children":2971},{},[2972,2985],{"type":40,"tag":125,"props":2973,"children":2974},{},[2975,2977,2983],{"type":46,"value":2976},"\"No ",{"type":40,"tag":67,"props":2978,"children":2980},{"className":2979},[],[2981],{"type":46,"value":2982},"vllm serve",{"type":46,"value":2984}," commands found\" \u002F \"page is a usage guide\":",{"type":46,"value":2986}," JS-rendering miss. recipes.vllm.ai pages always have ≥1 command. Ask the user to paste it or share the variant URL.",{"type":40,"tag":227,"props":2988,"children":2989},{},[2990,2995,2997,3003,3005,3011],{"type":40,"tag":125,"props":2991,"children":2992},{},[2993],{"type":46,"value":2994},"Single recipe returned",{"type":46,"value":2996}," for a model with known multiple variants → retry with variant-pinned URL. Axis names differ per model (Qwen: ",{"type":40,"tag":67,"props":2998,"children":3000},{"className":2999},[],[3001],{"type":46,"value":3002},"?variant=&strategy=",{"type":46,"value":3004},"; Kimi: ",{"type":40,"tag":67,"props":3006,"children":3008},{"className":3007},[],[3009],{"type":46,"value":3010},"?advanced=",{"type":46,"value":3012},"; others vary — no fixed pattern).",{"type":40,"tag":227,"props":3014,"children":3015},{},[3016,3021],{"type":40,"tag":125,"props":3017,"children":3018},{},[3019],{"type":46,"value":3020},"Variant label contradicts the command",{"type":46,"value":3022}," (e.g. label \"TEP\" but command shows DP+EP) → summarizer conflated variants; ask user.",{"type":40,"tag":49,"props":3024,"children":3025},{},[3026,3028,3033],{"type":46,"value":3027},"For non-trivial deployments (≥120B, multi-node, novel arch), ask the user which variant ",{"type":40,"tag":761,"props":3029,"children":3030},{},[3031],{"type":46,"value":3032},"before",{"type":46,"value":3034}," fetching.",{"type":40,"tag":49,"props":3036,"children":3037},{},[3038,3048],{"type":40,"tag":125,"props":3039,"children":3040},{},[3041,3043],{"type":46,"value":3042},"Source 2 — HF model card + ",{"type":40,"tag":67,"props":3044,"children":3046},{"className":3045},[],[3047],{"type":46,"value":2659},{"type":46,"value":3049}," (authoritative for):",{"type":40,"tag":1519,"props":3051,"children":3052},{},[3053,3068],{"type":40,"tag":1523,"props":3054,"children":3055},{},[3056],{"type":40,"tag":1527,"props":3057,"children":3058},{},[3059,3064],{"type":40,"tag":1531,"props":3060,"children":3061},{},[3062],{"type":46,"value":3063},"Signal",{"type":40,"tag":1531,"props":3065,"children":3066},{},[3067],{"type":46,"value":2848},{"type":40,"tag":1552,"props":3069,"children":3070},{},[3071,3090,3109,3135,3152],{"type":40,"tag":1527,"props":3072,"children":3073},{},[3074,3082],{"type":40,"tag":1559,"props":3075,"children":3076},{},[3077],{"type":40,"tag":67,"props":3078,"children":3080},{"className":3079},[],[3081],{"type":46,"value":2863},{"type":40,"tag":1559,"props":3083,"children":3084},{},[3085],{"type":40,"tag":67,"props":3086,"children":3088},{"className":3087},[],[3089],{"type":46,"value":2872},{"type":40,"tag":1527,"props":3091,"children":3092},{},[3093,3101],{"type":40,"tag":1559,"props":3094,"children":3095},{},[3096],{"type":40,"tag":67,"props":3097,"children":3099},{"className":3098},[],[3100],{"type":46,"value":2884},{"type":40,"tag":1559,"props":3102,"children":3103},{},[3104],{"type":40,"tag":67,"props":3105,"children":3107},{"className":3106},[],[3108],{"type":46,"value":2895},{"type":40,"tag":1527,"props":3110,"children":3111},{},[3112,3117],{"type":40,"tag":1559,"props":3113,"children":3114},{},[3115],{"type":46,"value":3116},"Reasoning\u002FCoT documented",{"type":40,"tag":1559,"props":3118,"children":3119},{},[3120,3125,3127,3133],{"type":40,"tag":67,"props":3121,"children":3123},{"className":3122},[],[3124],{"type":46,"value":2924},{"type":46,"value":3126}," (and ",{"type":40,"tag":67,"props":3128,"children":3130},{"className":3129},[],[3131],{"type":46,"value":3132},"--reasoning-parser-plugin",{"type":46,"value":3134}," if custom)",{"type":40,"tag":1527,"props":3136,"children":3137},{},[3138,3143],{"type":40,"tag":1559,"props":3139,"children":3140},{},[3141],{"type":46,"value":3142},"Tool-calling documented",{"type":40,"tag":1559,"props":3144,"children":3145},{},[3146],{"type":40,"tag":67,"props":3147,"children":3149},{"className":3148},[],[3150],{"type":46,"value":3151},"--enable-auto-tool-choice --tool-call-parser \u003Cparser>",{"type":40,"tag":1527,"props":3153,"children":3154},{},[3155,3160],{"type":40,"tag":1559,"props":3156,"children":3157},{},[3158],{"type":46,"value":3159},"Custom flags in card",{"type":40,"tag":1559,"props":3161,"children":3162},{},[3163,3165,3171],{"type":46,"value":3164},"Add as specified (e.g. ",{"type":40,"tag":67,"props":3166,"children":3168},{"className":3167},[],[3169],{"type":46,"value":3170},"--mamba_ssm_cache_dtype float32",{"type":46,"value":3172},")",{"type":40,"tag":49,"props":3174,"children":3175},{},[3176],{"type":40,"tag":125,"props":3177,"children":3178},{},[3179],{"type":46,"value":3180},"Cross-check rules:",{"type":40,"tag":223,"props":3182,"children":3183},{},[3184,3189,3194,3230],{"type":40,"tag":227,"props":3185,"children":3186},{},[3187],{"type":46,"value":3188},"Read both sources before composing the command.",{"type":40,"tag":227,"props":3190,"children":3191},{},[3192],{"type":46,"value":3193},"Agree → use with confidence.",{"type":40,"tag":227,"props":3195,"children":3196},{},[3197,3199,3204,3206,3212,3214,3220,3222,3228],{"type":46,"value":3198},"Disagree → ",{"type":40,"tag":125,"props":3200,"children":3201},{},[3202],{"type":46,"value":3203},"do not silently pick one.",{"type":46,"value":3205}," Surface both values to the user. Common conflicts: stale cards, parser rename between generations (Qwen2.5 ",{"type":40,"tag":67,"props":3207,"children":3209},{"className":3208},[],[3210],{"type":46,"value":3211},"hermes",{"type":46,"value":3213}," → Qwen3 ",{"type":40,"tag":67,"props":3215,"children":3217},{"className":3216},[],[3218],{"type":46,"value":3219},"qwen3_coder",{"type":46,"value":3221},"), recipe-only flags like ",{"type":40,"tag":67,"props":3223,"children":3225},{"className":3224},[],[3226],{"type":46,"value":3227},"--language-model-only",{"type":46,"value":3229},", ARM64-specific card notes.",{"type":40,"tag":227,"props":3231,"children":3232},{},[3233],{"type":46,"value":3234},"Resolve in Step 3 — don't defer to dry-run.",{"type":40,"tag":2897,"props":3236,"children":3238},{"id":3237},"vllm-deployment-command-structure-single-command-field",[3239,3241,3247],{"type":46,"value":3240},"vLLM deployment command structure — single ",{"type":40,"tag":67,"props":3242,"children":3244},{"className":3243},[],[3245],{"type":46,"value":3246},"command:",{"type":46,"value":3248}," field",{"type":40,"tag":49,"props":3250,"children":3251},{},[3252,3254,3259,3261,3267,3268,3274,3275,3281,3283,3289,3290,3296,3297,3303],{"type":46,"value":3253},"Rewrite the build-config output into one ",{"type":40,"tag":67,"props":3255,"children":3257},{"className":3256},[],[3258],{"type":46,"value":3246},{"type":46,"value":3260}," field. Move all parallelism (",{"type":40,"tag":67,"props":3262,"children":3264},{"className":3263},[],[3265],{"type":46,"value":3266},"--tensor-parallel-size",{"type":46,"value":303},{"type":40,"tag":67,"props":3269,"children":3271},{"className":3270},[],[3272],{"type":46,"value":3273},"--data-parallel-size",{"type":46,"value":303},{"type":40,"tag":67,"props":3276,"children":3278},{"className":3277},[],[3279],{"type":46,"value":3280},"--pipeline-parallel-size",{"type":46,"value":3282},") into the command; do not keep separate ",{"type":40,"tag":67,"props":3284,"children":3286},{"className":3285},[],[3287],{"type":46,"value":3288},"tensor_parallel_size",{"type":46,"value":744},{"type":40,"tag":67,"props":3291,"children":3293},{"className":3292},[],[3294],{"type":46,"value":3295},"data_parallel_size",{"type":46,"value":744},{"type":40,"tag":67,"props":3298,"children":3300},{"className":3299},[],[3301],{"type":46,"value":3302},"extra_args",{"type":46,"value":3304}," YAML fields.",{"type":40,"tag":98,"props":3306,"children":3310},{"className":3307,"code":3308,"language":3309,"meta":103,"style":103},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","deployment:\n  command: >-\n    vllm serve \u002Fcheckpoint\n    --host 0.0.0.0\n    --port ${deployment.port}\n    --tensor-parallel-size \u003CN>\n    --data-parallel-size \u003CM>\n    --max-model-len \u003Cvalue>\n    \u003C... rest of cross-checked flags ...>\n","yaml",[3311],{"type":40,"tag":67,"props":3312,"children":3313},{"__ignoreMap":103},[3314,3328,3352,3360,3368,3376,3385,3394,3403],{"type":40,"tag":2054,"props":3315,"children":3316},{"class":2056,"line":2057},[3317,3323],{"type":40,"tag":2054,"props":3318,"children":3320},{"style":3319},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[3321],{"type":46,"value":3322},"deployment",{"type":40,"tag":2054,"props":3324,"children":3325},{"style":2082},[3326],{"type":46,"value":3327},":\n",{"type":40,"tag":2054,"props":3329,"children":3330},{"class":2056,"line":2465},[3331,3336,3340,3346],{"type":40,"tag":2054,"props":3332,"children":3333},{"style":3319},[3334],{"type":46,"value":3335},"  command",{"type":40,"tag":2054,"props":3337,"children":3338},{"style":2082},[3339],{"type":46,"value":2829},{"type":40,"tag":2054,"props":3341,"children":3343},{"style":3342},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[3344],{"type":46,"value":3345}," >",{"type":40,"tag":2054,"props":3347,"children":3349},{"style":3348},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[3350],{"type":46,"value":3351},"-\n",{"type":40,"tag":2054,"props":3353,"children":3354},{"class":2056,"line":2474},[3355],{"type":40,"tag":2054,"props":3356,"children":3357},{"style":2066},[3358],{"type":46,"value":3359},"    vllm serve \u002Fcheckpoint\n",{"type":40,"tag":2054,"props":3361,"children":3362},{"class":2056,"line":2483},[3363],{"type":40,"tag":2054,"props":3364,"children":3365},{"style":2066},[3366],{"type":46,"value":3367},"    --host 0.0.0.0\n",{"type":40,"tag":2054,"props":3369,"children":3370},{"class":2056,"line":2492},[3371],{"type":40,"tag":2054,"props":3372,"children":3373},{"style":2066},[3374],{"type":46,"value":3375},"    --port ${deployment.port}\n",{"type":40,"tag":2054,"props":3377,"children":3379},{"class":2056,"line":3378},6,[3380],{"type":40,"tag":2054,"props":3381,"children":3382},{"style":2066},[3383],{"type":46,"value":3384},"    --tensor-parallel-size \u003CN>\n",{"type":40,"tag":2054,"props":3386,"children":3388},{"class":2056,"line":3387},7,[3389],{"type":40,"tag":2054,"props":3390,"children":3391},{"style":2066},[3392],{"type":46,"value":3393},"    --data-parallel-size \u003CM>\n",{"type":40,"tag":2054,"props":3395,"children":3397},{"class":2056,"line":3396},8,[3398],{"type":40,"tag":2054,"props":3399,"children":3400},{"style":2066},[3401],{"type":46,"value":3402},"    --max-model-len \u003Cvalue>\n",{"type":40,"tag":2054,"props":3404,"children":3406},{"class":2056,"line":3405},9,[3407],{"type":40,"tag":2054,"props":3408,"children":3409},{"style":2066},[3410],{"type":46,"value":3411},"    \u003C... rest of cross-checked flags ...>\n",{"type":40,"tag":49,"props":3413,"children":3414},{},[3415,3417,3423,3425,3431,3432,3437,3439,3444,3446,3452,3454,3460,3462,3468,3470,3475],{"type":46,"value":3416},"Conventions: always start ",{"type":40,"tag":67,"props":3418,"children":3420},{"className":3419},[],[3421],{"type":46,"value":3422},"vllm serve \u002Fcheckpoint",{"type":46,"value":3424}," (NEL mounts here); always ",{"type":40,"tag":67,"props":3426,"children":3428},{"className":3427},[],[3429],{"type":46,"value":3430},"--served-model-name ${deployment.served_model_name}",{"type":46,"value":736},{"type":40,"tag":125,"props":3433,"children":3434},{},[3435],{"type":46,"value":3436},"required",{"type":46,"value":3438},"; see ",{"type":40,"tag":67,"props":3440,"children":3442},{"className":3441},[],[3443],{"type":46,"value":1752},{"type":46,"value":3445}," for why); always ",{"type":40,"tag":67,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":46,"value":3451},"--host 0.0.0.0 --port ${deployment.port}",{"type":46,"value":3453},"; use folded scalar (",{"type":40,"tag":67,"props":3455,"children":3457},{"className":3456},[],[3458],{"type":46,"value":3459},">-",{"type":46,"value":3461},") for one flag per line. Example fallback ",{"type":40,"tag":67,"props":3463,"children":3465},{"className":3464},[],[3466],{"type":46,"value":3467},"--max-model-len 131072",{"type":46,"value":3469}," covers AA-LCR (~120K + 16K gen) and SciCode (≥ 65536) — prefer ",{"type":40,"tag":67,"props":3471,"children":3473},{"className":3472},[],[3474],{"type":46,"value":2659},{"type":46,"value":3476}," \u002F recipe value.",{"type":40,"tag":49,"props":3478,"children":3479},{},[3480,3482,3487,3488,3493,3494,3499,3501,3507,3509,3515],{"type":46,"value":3481},"For how to choose ",{"type":40,"tag":67,"props":3483,"children":3485},{"className":3484},[],[3486],{"type":46,"value":3266},{"type":46,"value":744},{"type":40,"tag":67,"props":3489,"children":3491},{"className":3490},[],[3492],{"type":46,"value":3273},{"type":46,"value":744},{"type":40,"tag":67,"props":3495,"children":3497},{"className":3496},[],[3498],{"type":46,"value":3280},{"type":46,"value":3500}," (and EP) from the model size and your GPU count, read ",{"type":40,"tag":67,"props":3502,"children":3504},{"className":3503},[],[3505],{"type":46,"value":3506},"references\u002Fparallelism.md",{"type":46,"value":3508}," — cross-check the layout against ",{"type":40,"tag":67,"props":3510,"children":3512},{"className":3511},[],[3513],{"type":46,"value":3514},"recipes.vllm.ai",{"type":46,"value":3516},", then adapt to the GPUs you actually have via the fit math there.",{"type":40,"tag":49,"props":3518,"children":3519},{},[3520,3525,3527,3533,3535,3540,3541,3546,3548,3554,3556,3562],{"type":40,"tag":125,"props":3521,"children":3522},{},[3523],{"type":46,"value":3524},"Image \u002F vLLM version.",{"type":46,"value":3526}," Treat default ",{"type":40,"tag":67,"props":3528,"children":3530},{"className":3529},[],[3531],{"type":46,"value":3532},"image: vllm\u002Fvllm-openai:v0.26.0",{"type":46,"value":3534}," as a floor to verify: bump to the ",{"type":40,"tag":125,"props":3536,"children":3537},{},[3538],{"type":46,"value":3539},"exact model's",{"type":46,"value":774},{"type":40,"tag":67,"props":3542,"children":3544},{"className":3543},[],[3545],{"type":46,"value":3514},{"type":46,"value":3547}," minimum if higher. Running below minimum is a trap — the server starts, then a worker dies mid-inference with ",{"type":40,"tag":67,"props":3549,"children":3551},{"className":3550},[],[3552],{"type":46,"value":3553},"CUDA error: an illegal memory access",{"type":46,"value":3555},", easy to misread as a kernel bug. A model newer than the latest release may have no numbered tag — use the image its recipe names. Never ",{"type":40,"tag":67,"props":3557,"children":3559},{"className":3558},[],[3560],{"type":46,"value":3561},":latest",{"type":46,"value":3563}," (breaks reproducibility). Surface version bumps to the user.",{"type":40,"tag":951,"props":3565,"children":3566},{},[3567,3589,3670],{"type":40,"tag":49,"props":3568,"children":3569},{},[3570,3575,3577,3583,3584],{"type":40,"tag":125,"props":3571,"children":3572},{},[3573],{"type":46,"value":3574},"NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 build",{"type":46,"value":3576}," — the cu12 build has no sm_103 FP4 kernel, so engine init dies with ",{"type":40,"tag":67,"props":3578,"children":3580},{"className":3579},[],[3581],{"type":46,"value":3582},"CUDA error: no kernel image is available",{"type":46,"value":535},{"type":40,"tag":125,"props":3585,"children":3586},{},[3587],{"type":46,"value":3588},"Pick the tag by the CUDA version it reports, not by its name — vLLM inverted its tag convention at v0.20.0:",{"type":40,"tag":1519,"props":3590,"children":3591},{},[3592,3613],{"type":40,"tag":1523,"props":3593,"children":3594},{},[3595],{"type":40,"tag":1527,"props":3596,"children":3597},{},[3598,3603,3608],{"type":40,"tag":1531,"props":3599,"children":3600},{},[3601],{"type":46,"value":3602},"vLLM version",{"type":40,"tag":1531,"props":3604,"children":3605},{},[3606],{"type":46,"value":3607},"CUDA-13 tag",{"type":40,"tag":1531,"props":3609,"children":3610},{},[3611],{"type":46,"value":3612},"CUDA-12 tag",{"type":40,"tag":1552,"props":3614,"children":3615},{},[3616,3644],{"type":40,"tag":1527,"props":3617,"children":3618},{},[3619,3624,3639],{"type":40,"tag":1559,"props":3620,"children":3621},{},[3622],{"type":46,"value":3623},"≤ v0.19.x",{"type":40,"tag":1559,"props":3625,"children":3626},{},[3627,3632,3633],{"type":40,"tag":125,"props":3628,"children":3629},{},[3630],{"type":46,"value":3631},"suffixed",{"type":46,"value":774},{"type":40,"tag":67,"props":3634,"children":3636},{"className":3635},[],[3637],{"type":46,"value":3638},"-cu130",{"type":40,"tag":1559,"props":3640,"children":3641},{},[3642],{"type":46,"value":3643},"unsuffixed",{"type":40,"tag":1527,"props":3645,"children":3646},{},[3647,3652,3659],{"type":40,"tag":1559,"props":3648,"children":3649},{},[3650],{"type":46,"value":3651},"≥ v0.20.0",{"type":40,"tag":1559,"props":3653,"children":3654},{},[3655],{"type":40,"tag":125,"props":3656,"children":3657},{},[3658],{"type":46,"value":3643},{"type":40,"tag":1559,"props":3660,"children":3661},{},[3662,3664],{"type":46,"value":3663},"suffixed ",{"type":40,"tag":67,"props":3665,"children":3667},{"className":3666},[],[3668],{"type":46,"value":3669},"-cu129",{"type":40,"tag":49,"props":3671,"children":3672},{},[3673,3675,3680,3682,3695,3697,3702,3704,3710,3712,3718,3720,3725],{"type":46,"value":3674},"v0.20.0 ships both suffixes; after it ",{"type":40,"tag":67,"props":3676,"children":3678},{"className":3677},[],[3679],{"type":46,"value":3638},{"type":46,"value":3681}," doesn't exist, so asking for it yields a missing tag. ",{"type":40,"tag":125,"props":3683,"children":3684},{},[3685,3687,3693],{"type":46,"value":3686},"Select a tag whose config blob reports ",{"type":40,"tag":67,"props":3688,"children":3690},{"className":3689},[],[3691],{"type":46,"value":3692},"CUDA_VERSION",{"type":46,"value":3694}," ≥ 13",{"type":46,"value":3696}," (registry API), reading the child manifest for ",{"type":40,"tag":125,"props":3698,"children":3699},{},[3700],{"type":46,"value":3701},"the platform you deploy on",{"type":46,"value":3703}," (arm64 Grace\u002FGB300, amd64 x86) — ",{"type":40,"tag":67,"props":3705,"children":3707},{"className":3706},[],[3708],{"type":46,"value":3709},"TORCH_CUDA_ARCH_LIST",{"type":46,"value":3711}," differs per platform, so check your arch against that child. Multimodal on sm_103 may also need ",{"type":40,"tag":67,"props":3713,"children":3715},{"className":3714},[],[3716],{"type":46,"value":3717},"--mm-encoder-attn-backend TRITON_ATTN",{"type":46,"value":3719},". Full note in ",{"type":40,"tag":67,"props":3721,"children":3723},{"className":3722},[],[3724],{"type":46,"value":1715},{"type":46,"value":258},{"type":40,"tag":2897,"props":3727,"children":3729},{"id":3728},"vllm-backend-defaults-always-include-unless-the-recipe-contradicts",[3730,3732],{"type":46,"value":3731},"vLLM-backend defaults — always include unless the recipe ",{"type":40,"tag":761,"props":3733,"children":3734},{},[3735],{"type":46,"value":3736},"contradicts",{"type":40,"tag":49,"props":3738,"children":3739},{},[3740,3742,3748],{"type":46,"value":3741},"Silence is not contradiction. Drop\u002Foverride only when the recipe sets a different value for the same setting (e.g. recipe pins ",{"type":40,"tag":67,"props":3743,"children":3745},{"className":3744},[],[3746],{"type":46,"value":3747},"--max-num-batched-tokens 16384",{"type":46,"value":3749}," → use 16384).",{"type":40,"tag":1329,"props":3751,"children":3752},{},[3753,3792,3803,3814,3881],{"type":40,"tag":227,"props":3754,"children":3755},{},[3756,3762,3763,3768,3770,3775,3777,3783,3785,3790],{"type":40,"tag":67,"props":3757,"children":3759},{"className":3758},[],[3760],{"type":46,"value":3761},"--model-loader-extra-config '{\"enable_multithread_load\": true, \"num_threads\": 128}'",{"type":46,"value":1446},{"type":40,"tag":125,"props":3764,"children":3765},{},[3766],{"type":46,"value":3767},"parallelizes checkpoint load",{"type":46,"value":3769},", the single biggest deploy-time cost for large checkpoints. A big MoE otherwise loads shards ~sequentially (~1 min\u002Fshard → e.g. ~40 min for a ~450 GB \u002F 45-shard checkpoint); on a ",{"type":40,"tag":125,"props":3771,"children":3772},{},[3773],{"type":46,"value":3774},"preemptible",{"type":46,"value":3776}," queue that long load window is exactly where jobs get killed before they ever serve. ",{"type":40,"tag":67,"props":3778,"children":3780},{"className":3779},[],[3781],{"type":46,"value":3782},"num_threads",{"type":46,"value":3784}," defaults to ",{"type":40,"tag":125,"props":3786,"children":3787},{},[3788],{"type":46,"value":3789},"128",{"type":46,"value":3791},"; scale it to the checkpoint (smaller for small models, bounded by the shared-FS read bandwidth — too high yields no gain). Safe to always include.",{"type":40,"tag":227,"props":3793,"children":3794},{},[3795,3801],{"type":40,"tag":67,"props":3796,"children":3798},{"className":3797},[],[3799],{"type":46,"value":3800},"--max-num-batched-tokens 8192",{"type":46,"value":3802}," — caps per-step batched tokens; prevents long-prefill stalls.",{"type":40,"tag":227,"props":3804,"children":3805},{},[3806,3812],{"type":40,"tag":67,"props":3807,"children":3809},{"className":3808},[],[3810],{"type":46,"value":3811},"--enable-chunked-prefill",{"type":46,"value":3813}," — interleaves long prefills with decode steps (required for AA-LCR's ~120K input). Modern vLLM defaults this on for many models; set explicitly to avoid drift.",{"type":40,"tag":227,"props":3815,"children":3816},{},[3817,3823,3824,3829,3831,3837,3838,3844,3846,3852,3853,3859,3860,3866,3867,3872,3874,3879],{"type":40,"tag":67,"props":3818,"children":3820},{"className":3819},[],[3821],{"type":46,"value":3822},"--enable-expert-parallel",{"type":46,"value":1446},{"type":40,"tag":125,"props":3825,"children":3826},{},[3827],{"type":46,"value":3828},"MoE-only default.",{"type":46,"value":3830}," Detect MoE from handle suffix (",{"type":40,"tag":67,"props":3832,"children":3834},{"className":3833},[],[3835],{"type":46,"value":3836},"-A10B",{"type":46,"value":303},{"type":40,"tag":67,"props":3839,"children":3841},{"className":3840},[],[3842],{"type":46,"value":3843},"-A3B",{"type":46,"value":3845},", etc.), ",{"type":40,"tag":67,"props":3847,"children":3849},{"className":3848},[],[3850],{"type":46,"value":3851},"num_experts",{"type":46,"value":744},{"type":40,"tag":67,"props":3854,"children":3856},{"className":3855},[],[3857],{"type":46,"value":3858},"num_local_experts",{"type":46,"value":744},{"type":40,"tag":67,"props":3861,"children":3863},{"className":3862},[],[3864],{"type":46,"value":3865},"n_routed_experts",{"type":46,"value":488},{"type":40,"tag":67,"props":3868,"children":3870},{"className":3869},[],[3871],{"type":46,"value":2659},{"type":46,"value":3873},", or card. No-op when TP=DP=1, safe to always include for MoE. Do not add for dense models. See ",{"type":40,"tag":67,"props":3875,"children":3877},{"className":3876},[],[3878],{"type":46,"value":3506},{"type":46,"value":3880}," for what EP does and the DP-attention + EP-MoE throughput pattern.",{"type":40,"tag":227,"props":3882,"children":3883},{},[3884,3890,3891,3896,3898,3904,3905,3910,3912,3917,3918,3927,3931,3933,3939,3941,3946,3948,3953],{"type":40,"tag":67,"props":3885,"children":3887},{"className":3886},[],[3888],{"type":46,"value":3889},"--max-num-seqs N",{"type":46,"value":1446},{"type":40,"tag":125,"props":3892,"children":3893},{},[3894],{"type":46,"value":3895},"omit at generation time",{"type":46,"value":3897}," (top-level ",{"type":40,"tag":67,"props":3899,"children":3901},{"className":3900},[],[3902],{"type":46,"value":3903},"parallelism",{"type":46,"value":868},{"type":40,"tag":67,"props":3906,"children":3908},{"className":3907},[],[3909],{"type":46,"value":874},{"type":46,"value":3911},"). Add this comment above ",{"type":40,"tag":67,"props":3913,"children":3915},{"className":3914},[],[3916],{"type":46,"value":3246},{"type":46,"value":2829},{"type":40,"tag":98,"props":3919,"children":3922},{"className":3920,"code":3921,"language":46,"meta":103},[101],"# After filling in `parallelism` values (top-level + per-task overrides),\n# append `--max-num-seqs N` where N = ceil(max_parallelism \u002F data_parallel_size).\n",[3923],{"type":40,"tag":67,"props":3924,"children":3925},{"__ignoreMap":103},[3926],{"type":46,"value":3921},{"type":40,"tag":3928,"props":3929,"children":3930},"br",{},[],{"type":46,"value":3932},"In Step 4 compute and append. Example: top-level=16, Tau2=128, DP=8 → ",{"type":40,"tag":67,"props":3934,"children":3936},{"className":3935},[],[3937],{"type":46,"value":3938},"ceil(128\u002F8)=16",{"type":46,"value":3940},". Too small → request queuing; too large → wasted KV reservation. For how to choose the ",{"type":40,"tag":67,"props":3942,"children":3944},{"className":3943},[],[3945],{"type":46,"value":3903},{"type":46,"value":3947}," it derives from, read ",{"type":40,"tag":67,"props":3949,"children":3951},{"className":3950},[],[3952],{"type":46,"value":3506},{"type":46,"value":258},{"type":40,"tag":2897,"props":3955,"children":3957},{"id":3956},"evaluation-params-template-top-level-params",[3958],{"type":46,"value":3959},"Evaluation params template (top-level params)",{"type":40,"tag":49,"props":3961,"children":3962},{},[3963,3965,3971,3973,3978,3980,3986,3987,3993,3994,4000,4001,4007],{"type":46,"value":3964},"The top-level ",{"type":40,"tag":67,"props":3966,"children":3968},{"className":3967},[],[3969],{"type":46,"value":3970},"nemo_evaluator_config.config.params",{"type":46,"value":3972}," must contain ",{"type":40,"tag":125,"props":3974,"children":3975},{},[3976],{"type":46,"value":3977},"exactly these six fields",{"type":46,"value":3979}," — no ",{"type":40,"tag":67,"props":3981,"children":3983},{"className":3982},[],[3984],{"type":46,"value":3985},"top_k",{"type":46,"value":744},{"type":40,"tag":67,"props":3988,"children":3990},{"className":3989},[],[3991],{"type":46,"value":3992},"presence_penalty",{"type":46,"value":744},{"type":40,"tag":67,"props":3995,"children":3997},{"className":3996},[],[3998],{"type":46,"value":3999},"repetition_penalty",{"type":46,"value":744},{"type":40,"tag":67,"props":4002,"children":4004},{"className":4003},[],[4005],{"type":46,"value":4006},"min_p",{"type":46,"value":2829},{"type":40,"tag":98,"props":4009,"children":4011},{"className":3307,"code":4010,"language":3309,"meta":103,"style":103},"nemo_evaluator_config:\n  config:\n    params:\n      parallelism: ???    # Required — size per references\u002Fparallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear\n      request_timeout: 3600\n      max_retries: 10\n      max_new_tokens: 65536  # see rule below\n      temperature: 1.0    # from model card (reasoning); adjust\n      top_p: 0.95         # from model card (reasoning); adjust\n",[4012],{"type":40,"tag":67,"props":4013,"children":4014},{"__ignoreMap":103},[4015,4027,4039,4051,4074,4092,4109,4131,4153],{"type":40,"tag":2054,"props":4016,"children":4017},{"class":2056,"line":2057},[4018,4023],{"type":40,"tag":2054,"props":4019,"children":4020},{"style":3319},[4021],{"type":46,"value":4022},"nemo_evaluator_config",{"type":40,"tag":2054,"props":4024,"children":4025},{"style":2082},[4026],{"type":46,"value":3327},{"type":40,"tag":2054,"props":4028,"children":4029},{"class":2056,"line":2465},[4030,4035],{"type":40,"tag":2054,"props":4031,"children":4032},{"style":3319},[4033],{"type":46,"value":4034},"  config",{"type":40,"tag":2054,"props":4036,"children":4037},{"style":2082},[4038],{"type":46,"value":3327},{"type":40,"tag":2054,"props":4040,"children":4041},{"class":2056,"line":2474},[4042,4047],{"type":40,"tag":2054,"props":4043,"children":4044},{"style":3319},[4045],{"type":46,"value":4046},"    params",{"type":40,"tag":2054,"props":4048,"children":4049},{"style":2082},[4050],{"type":46,"value":3327},{"type":40,"tag":2054,"props":4052,"children":4053},{"class":2056,"line":2483},[4054,4059,4063,4068],{"type":40,"tag":2054,"props":4055,"children":4056},{"style":3319},[4057],{"type":46,"value":4058},"      parallelism",{"type":40,"tag":2054,"props":4060,"children":4061},{"style":2082},[4062],{"type":46,"value":2829},{"type":40,"tag":2054,"props":4064,"children":4065},{"style":2093},[4066],{"type":46,"value":4067}," ???    ",{"type":40,"tag":2054,"props":4069,"children":4071},{"style":4070},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[4072],{"type":46,"value":4073},"# Required — size per references\u002Fparallelism.md (bounded by total request count vs GPU serving capacity); ask user in Step 4 if still unclear\n",{"type":40,"tag":2054,"props":4075,"children":4076},{"class":2056,"line":2492},[4077,4082,4086],{"type":40,"tag":2054,"props":4078,"children":4079},{"style":3319},[4080],{"type":46,"value":4081},"      request_timeout",{"type":40,"tag":2054,"props":4083,"children":4084},{"style":2093},[4085],{"type":46,"value":799},{"type":40,"tag":2054,"props":4087,"children":4089},{"style":4088},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[4090],{"type":46,"value":4091},"3600\n",{"type":40,"tag":2054,"props":4093,"children":4094},{"class":2056,"line":3378},[4095,4100,4104],{"type":40,"tag":2054,"props":4096,"children":4097},{"style":3319},[4098],{"type":46,"value":4099},"      max_retries",{"type":40,"tag":2054,"props":4101,"children":4102},{"style":2082},[4103],{"type":46,"value":2829},{"type":40,"tag":2054,"props":4105,"children":4106},{"style":4088},[4107],{"type":46,"value":4108}," 10\n",{"type":40,"tag":2054,"props":4110,"children":4111},{"class":2056,"line":3387},[4112,4117,4121,4126],{"type":40,"tag":2054,"props":4113,"children":4114},{"style":3319},[4115],{"type":46,"value":4116},"      max_new_tokens",{"type":40,"tag":2054,"props":4118,"children":4119},{"style":2082},[4120],{"type":46,"value":2829},{"type":40,"tag":2054,"props":4122,"children":4123},{"style":4088},[4124],{"type":46,"value":4125}," 65536",{"type":40,"tag":2054,"props":4127,"children":4128},{"style":4070},[4129],{"type":46,"value":4130},"  # see rule below\n",{"type":40,"tag":2054,"props":4132,"children":4133},{"class":2056,"line":3396},[4134,4139,4143,4148],{"type":40,"tag":2054,"props":4135,"children":4136},{"style":3319},[4137],{"type":46,"value":4138},"      temperature",{"type":40,"tag":2054,"props":4140,"children":4141},{"style":2082},[4142],{"type":46,"value":2829},{"type":40,"tag":2054,"props":4144,"children":4145},{"style":4088},[4146],{"type":46,"value":4147}," 1.0",{"type":40,"tag":2054,"props":4149,"children":4150},{"style":4070},[4151],{"type":46,"value":4152},"    # from model card (reasoning); adjust\n",{"type":40,"tag":2054,"props":4154,"children":4155},{"class":2056,"line":3405},[4156,4161,4165,4170],{"type":40,"tag":2054,"props":4157,"children":4158},{"style":3319},[4159],{"type":46,"value":4160},"      top_p",{"type":40,"tag":2054,"props":4162,"children":4163},{"style":2082},[4164],{"type":46,"value":2829},{"type":40,"tag":2054,"props":4166,"children":4167},{"style":4088},[4168],{"type":46,"value":4169}," 0.95",{"type":40,"tag":2054,"props":4171,"children":4172},{"style":4070},[4173],{"type":46,"value":4174},"         # from model card (reasoning); adjust\n",{"type":40,"tag":49,"props":4176,"children":4177},{},[4178,4180,4185],{"type":46,"value":4179},"Per-task ",{"type":40,"tag":67,"props":4181,"children":4183},{"className":4182},[],[4184],{"type":46,"value":1889},{"type":46,"value":4186}," overrides are forbidden — set one top-level ceiling everywhere.",{"type":40,"tag":49,"props":4188,"children":4189},{},[4190,4220,4222,4228,4230,4236,4237,4243,4245,4250,4252,4257],{"type":40,"tag":125,"props":4191,"children":4192},{},[4193,4195,4200,4201,4206,4207,4212,4214],{"type":46,"value":4194},"Cross-check ",{"type":40,"tag":67,"props":4196,"children":4198},{"className":4197},[],[4199],{"type":46,"value":1875},{"type":46,"value":744},{"type":40,"tag":67,"props":4202,"children":4204},{"className":4203},[],[4205],{"type":46,"value":1882},{"type":46,"value":744},{"type":40,"tag":67,"props":4208,"children":4210},{"className":4209},[],[4211],{"type":46,"value":1889},{"type":46,"value":4213}," against ",{"type":40,"tag":67,"props":4215,"children":4217},{"className":4216},[],[4218],{"type":46,"value":4219},"references\u002Fnvfp4-modelcard-sampling.md",{"type":46,"value":4221}," — the published settings for the 2026 NVFP4 checkpoints under ",{"type":40,"tag":67,"props":4223,"children":4225},{"className":4224},[],[4226],{"type":46,"value":4227},"huggingface.co\u002Fnvidia",{"type":46,"value":4229}," that disclose them (older releases and cards that publish nothing are absent — for those, read the card; ",{"type":40,"tag":67,"props":4231,"children":4233},{"className":4232},[],[4234],{"type":46,"value":4235},"-DSpark",{"type":46,"value":744},{"type":40,"tag":67,"props":4238,"children":4240},{"className":4239},[],[4241],{"type":46,"value":4242},"-DFlash",{"type":46,"value":4244}," spec-decode variants share their base checkpoint's row, since spec decoding does not change the target's output distribution). ",{"type":40,"tag":125,"props":4246,"children":4247},{},[4248],{"type":46,"value":4249},"The card is the source of truth; this file is a reference, not a constraint",{"type":46,"value":4251}," — use it to confirm a value you read, to fill a gap when the card is silent or ambiguous, and to catch a misreading. Worth consulting whenever the model is an NVFP4 checkpoint ",{"type":40,"tag":125,"props":4253,"children":4254},{},[4255],{"type":46,"value":4256},"or shares a family with one",{"type":46,"value":4258}," (Qwen3.x, GLM-4.7\u002F5.x, Kimi K2.x\u002FK3, MiniMax M2.x\u002FM3, DeepSeek V3.x\u002FV4\u002FR1, Gemma 4, Nemotron 3\u002F3.5, Llama-Nemotron, Mistral Medium 3.5), and especially when you are unsure. It is a dated snapshot, so for anything newer than it, trust the card. See that file's \"Lookup\" section.",{"type":40,"tag":49,"props":4260,"children":4261},{},[4262,4278,4280,4286,4288,4294,4296,4302,4304,4316,4318,4323],{"type":40,"tag":125,"props":4263,"children":4264},{},[4265,4270,4271,4276],{"type":40,"tag":67,"props":4266,"children":4268},{"className":4267},[],[4269],{"type":46,"value":1875},{"type":46,"value":744},{"type":40,"tag":67,"props":4272,"children":4274},{"className":4273},[],[4275],{"type":46,"value":1882},{"type":46,"value":4277}," are different: per-task overrides ARE allowed and often required.",{"type":46,"value":4279}," Cards often specify sampling per scenario — DeepSeek-V4-Pro-0813 gives ",{"type":40,"tag":67,"props":4281,"children":4283},{"className":4282},[],[4284],{"type":46,"value":4285},"top_p = 0.95",{"type":46,"value":4287}," for agentic scenarios and ",{"type":40,"tag":67,"props":4289,"children":4291},{"className":4290},[],[4292],{"type":46,"value":4293},"1.0",{"type":46,"value":4295}," otherwise, so a single top-level ",{"type":40,"tag":67,"props":4297,"children":4299},{"className":4298},[],[4300],{"type":46,"value":4301},"0.95",{"type":46,"value":4303}," is wrong for every non-agentic task.\nSet the top-level value for the majority case, override only the tasks the card calls out, and apply\nthe split identically to baseline and candidate. ",{"type":40,"tag":125,"props":4305,"children":4306},{},[4307,4309,4314],{"type":46,"value":4308},"The ",{"type":40,"tag":67,"props":4310,"children":4312},{"className":4311},[],[4313],{"type":46,"value":1780},{"type":46,"value":4315}," tags record only the\ntop-level values",{"type":46,"value":4317},", so note any per-task override in the run ",{"type":40,"tag":67,"props":4319,"children":4321},{"className":4320},[],[4322],{"type":46,"value":1809},{"type":46,"value":4324}," — otherwise the\noverridden task is reported under sampling params it did not use.",{"type":40,"tag":2897,"props":4326,"children":4328},{"id":4327},"max_new_tokens-mandatory-model-card-lookup",[4329,4334],{"type":40,"tag":67,"props":4330,"children":4332},{"className":4331},[],[4333],{"type":46,"value":1889},{"type":46,"value":4335}," — mandatory model-card lookup",{"type":40,"tag":223,"props":4337,"children":4338},{},[4339,4349,4413,4452,4477,4495],{"type":40,"tag":227,"props":4340,"children":4341},{},[4342,4347],{"type":40,"tag":125,"props":4343,"children":4344},{},[4345],{"type":46,"value":4346},"Fetch the HF model card before writing the value.",{"type":46,"value":4348}," Not optional.",{"type":40,"tag":227,"props":4350,"children":4351},{},[4352,4354,4360,4361,4366,4368,4373,4375,4380,4382,4387,4389,4395,4397,4403,4405,4411],{"type":46,"value":4353},"Scan for any ",{"type":40,"tag":67,"props":4355,"children":4357},{"className":4356},[],[4358],{"type":46,"value":4359},"max_tokens",{"type":46,"value":744},{"type":40,"tag":67,"props":4362,"children":4364},{"className":4363},[],[4365],{"type":46,"value":1889},{"type":46,"value":4367}," \u002F \"output length\" recommendation. Pick the ",{"type":40,"tag":125,"props":4369,"children":4370},{},[4371],{"type":46,"value":4372},"highest",{"type":46,"value":4374}," value the card mentions (Qwen3.6: 32768 general + 81920 math-coding → use ",{"type":40,"tag":125,"props":4376,"children":4377},{},[4378],{"type":46,"value":4379},"81920",{"type":46,"value":4381},"). Annotate with a citing comment.\n",{"type":40,"tag":125,"props":4383,"children":4384},{},[4385],{"type":46,"value":4386},"Card figures are SINGLE-TURN.",{"type":46,"value":4388}," On multi-turn \u002F agentic benchmarks the model's own answer is fed back in, so the cap must satisfy ",{"type":40,"tag":67,"props":4390,"children":4392},{"className":4391},[],[4393],{"type":46,"value":4394},"n_turns × max_new_tokens + prompt \u003C max_model_len",{"type":46,"value":4396},". Taking a card's headline \"384K output\" literally lost SciCode samples to HTTP 400; 65536 was clean. (",{"type":40,"tag":67,"props":4398,"children":4400},{"className":4399},[],[4401],{"type":46,"value":4402},"references\u002Frun-validation.md",{"type":46,"value":4404}," already covers checking ",{"type":40,"tag":67,"props":4406,"children":4408},{"className":4407},[],[4409],{"type":46,"value":4410},"finish_reason: length",{"type":46,"value":4412}," after a run.)",{"type":40,"tag":227,"props":4414,"children":4415},{},[4416,4428,4430,4435,4437,4443,4445,4450],{"type":40,"tag":125,"props":4417,"children":4418},{},[4419,4421,4426],{"type":46,"value":4420},"Consult ",{"type":40,"tag":67,"props":4422,"children":4424},{"className":4423},[],[4425],{"type":46,"value":4219},{"type":46,"value":4427}," as a reference.",{"type":46,"value":4429}," Listed and in agreement → proceed with confidence. Listed and different → ",{"type":40,"tag":125,"props":4431,"children":4432},{},[4433],{"type":46,"value":4434},"the card wins",{"type":46,"value":4436},"; re-read it, then note the discrepancy for the user rather than auto-correcting either way. Not listed, or the card is silent or ambiguous → take the nearest same-family rows as the value, a far better prior than the generic fallback below. Its ",{"type":40,"tag":67,"props":4438,"children":4440},{"className":4439},[],[4441],{"type":46,"value":4442},"max_num_tokens",{"type":46,"value":4444}," column records the card's ",{"type":40,"tag":761,"props":4446,"children":4447},{},[4448],{"type":46,"value":4449},"headline",{"type":46,"value":4451}," cap, so rule 2 above still governs: when a card names more than one cap, the highest wins even if that exceeds the row.",{"type":40,"tag":227,"props":4453,"children":4454},{},[4455,4457,4461,4463,4468,4470,4475],{"type":46,"value":4456},"If the card is genuinely silent after a thorough read ",{"type":40,"tag":125,"props":4458,"children":4459},{},[4460],{"type":46,"value":765},{"type":46,"value":4462}," the family table offers no usable pattern, fall back to: ",{"type":40,"tag":125,"props":4464,"children":4465},{},[4466],{"type":46,"value":4467},"65536",{"type":46,"value":4469}," (reasoning), ",{"type":40,"tag":125,"props":4471,"children":4472},{},[4473],{"type":46,"value":4474},"16384",{"type":46,"value":4476}," (non-reasoning); surface the silence to the user.",{"type":40,"tag":227,"props":4478,"children":4479},{},[4480,4485,4487,4493],{"type":40,"tag":125,"props":4481,"children":4482},{},[4483],{"type":46,"value":4484},"Forbidden:",{"type":46,"value":4486}," writing ",{"type":40,"tag":67,"props":4488,"children":4490},{"className":4489},[],[4491],{"type":46,"value":4492},"max_new_tokens: \u003Cgeneric_default>",{"type":46,"value":4494}," with a \"card not yet checked\" comment. Either fetch and apply, or fetch and confirm silence.",{"type":40,"tag":227,"props":4496,"children":4497},{},[4498,4503,4505,4511],{"type":40,"tag":125,"props":4499,"children":4500},{},[4501],{"type":46,"value":4502},"A higher cap doesn't fix runaway reasoning.",{"type":46,"value":4504}," On hard tasks (e.g. HLE) a non-terminating model just rambles to the larger cap (~80% length-capped at 131072), and the cap only helps if deployment ",{"type":40,"tag":67,"props":4506,"children":4508},{"className":4507},[],[4509],{"type":46,"value":4510},"--max-model-len > prompt + max_new_tokens",{"type":46,"value":4512}," (else generation is silently clipped — AA-LCR's ~120K input leaves little room). Treat such tasks as low-confidence.",{"type":40,"tag":2897,"props":4514,"children":4516},{"id":4515},"quantization-aware-benchmark-defaults",[4517],{"type":46,"value":4518},"Quantization-aware benchmark defaults",{"type":40,"tag":49,"props":4520,"children":4521},{},[4522,4524,4529,4531,4537,4539,4544],{"type":46,"value":4523},"For quantized checkpoints, read ",{"type":40,"tag":67,"props":4525,"children":4527},{"className":4526},[],[4528],{"type":46,"value":1342},{"type":46,"value":4530}," for sensitivity rankings and recommended sets; present and ask which to include. Read ",{"type":40,"tag":67,"props":4532,"children":4534},{"className":4533},[],[4535],{"type":46,"value":4536},"references\u002Fmodel-card-research.md",{"type":46,"value":4538}," for the full extraction checklist (sampling, reasoning config, ARM64, ",{"type":40,"tag":67,"props":4540,"children":4542},{"className":4541},[],[4543],{"type":46,"value":592},{"type":46,"value":4545},", output length — see the dedicated bullet there).",{"type":40,"tag":49,"props":4547,"children":4548},{},[4549],{"type":46,"value":4550},"Reasoning models: prefer reasoning mode (highest scores). For lower variance \u002F cost \u002F apples-to-apples vs non-reasoning baselines, also consider a non-reasoning companion run.",{"type":40,"tag":2897,"props":4552,"children":4554},{"id":4553},"reasoning-adapter-config-use_reasoning",[4555,4557,4563],{"type":46,"value":4556},"Reasoning adapter config (",{"type":40,"tag":67,"props":4558,"children":4560},{"className":4559},[],[4561],{"type":46,"value":4562},"use_reasoning",{"type":46,"value":3172},{"type":40,"tag":49,"props":4565,"children":4566},{},[4567,4568,4574,4576,4581,4583,4589],{"type":46,"value":4308},{"type":40,"tag":67,"props":4569,"children":4571},{"className":4570},[],[4572],{"type":46,"value":4573},"adapter_config",{"type":46,"value":4575}," block in ",{"type":40,"tag":67,"props":4577,"children":4579},{"className":4578},[],[4580],{"type":46,"value":1752},{"type":46,"value":4582}," controls request\u002Fresponse\nlogging and reasoning handling. ",{"type":40,"tag":67,"props":4584,"children":4586},{"className":4585},[],[4587],{"type":46,"value":4588},"use_reasoning: true",{"type":46,"value":4590}," strips the model's\nreasoning\u002FCoT trace before scoring (grade only the final answer). Set per type:",{"type":40,"tag":223,"props":4592,"children":4593},{},[4594,4618,4640],{"type":40,"tag":227,"props":4595,"children":4596},{},[4597,4608,4610,4616],{"type":40,"tag":125,"props":4598,"children":4599},{},[4600,4602],{"type":46,"value":4601},"Instruct → ",{"type":40,"tag":67,"props":4603,"children":4605},{"className":4604},[],[4606],{"type":46,"value":4607},"use_reasoning: false",{"type":46,"value":4609}," and drop the ",{"type":40,"tag":67,"props":4611,"children":4613},{"className":4612},[],[4614],{"type":46,"value":4615},"chat_template_kwargs",{"type":46,"value":4617},"\nthinking block (no trace to strip; can mangle plain responses).",{"type":40,"tag":227,"props":4619,"children":4620},{},[4621,4631,4633,4638],{"type":40,"tag":125,"props":4622,"children":4623},{},[4624,4626],{"type":46,"value":4625},"Reasoning → ",{"type":40,"tag":67,"props":4627,"children":4629},{"className":4628},[],[4630],{"type":46,"value":4588},{"type":46,"value":4632},", especially when the deployment sets\n",{"type":40,"tag":67,"props":4634,"children":4636},{"className":4635},[],[4637],{"type":46,"value":2924},{"type":46,"value":4639}," (vLLM emits a separate reasoning channel to strip).",{"type":40,"tag":227,"props":4641,"children":4642},{},[4643,4655,4656,4661,4663,4668,4670,4675],{"type":40,"tag":125,"props":4644,"children":4645},{},[4646,4648,4653],{"type":46,"value":4647},"Hybrid (reasoning on ",{"type":40,"tag":761,"props":4649,"children":4650},{},[4651],{"type":46,"value":4652},"or",{"type":46,"value":4654}," off) → turn it ON",{"type":46,"value":736},{"type":40,"tag":67,"props":4657,"children":4659},{"className":4658},[],[4660],{"type":46,"value":4588},{"type":46,"value":4662}," +\nforce the thinking flag in ",{"type":40,"tag":67,"props":4664,"children":4666},{"className":4665},[],[4667],{"type":46,"value":4615},{"type":46,"value":4669},"). For the exact toggle key\n(it drifts across generations) and the reasoning-effort policy, see\n",{"type":40,"tag":67,"props":4671,"children":4673},{"className":4672},[],[4674],{"type":46,"value":4536},{"type":46,"value":4676}," → \"Reasoning config\".",{"type":40,"tag":110,"props":4678,"children":4679},{},[],{"type":40,"tag":55,"props":4681,"children":4683},{"id":4682},"step-4-fill-remaining-values",[4684],{"type":46,"value":4685},"Step 4 — Fill remaining ??? values",{"type":40,"tag":49,"props":4687,"children":4688},{},[4689,4694,4696,4702,4704,4710],{"type":40,"tag":125,"props":4690,"children":4691},{},[4692],{"type":46,"value":4693},"Predefined per-cluster execution config (check FIRST).",{"type":46,"value":4695}," Some installs ship ",{"type":40,"tag":67,"props":4697,"children":4699},{"className":4698},[],[4700],{"type":46,"value":4701},"internal\u002Fslurm\u002F\u003Ccluster>",{"type":46,"value":4703}," execution groups (optional ",{"type":40,"tag":67,"props":4705,"children":4707},{"className":4706},[],[4708],{"type":46,"value":4709},"nemo_evaluator_launcher_internal",{"type":46,"value":4711}," pkg) that pre-fill hostname\u002Fpartition\u002Fgres — leaving only account\u002Foutput_dir\u002Fwalltime. Discover at runtime (nothing cluster-specific hardcoded):",{"type":40,"tag":98,"props":4713,"children":4715},{"className":2046,"code":4714,"language":2048,"meta":103,"style":103},"python3 -c 'import nemo_evaluator_launcher_internal' 2>\u002Fdev\u002Fnull && \\\nPKG=$(python3 -c 'import nemo_evaluator_launcher_internal as m,os;print(os.path.dirname(m.__file__))') && \\\nfor f in \"$PKG\"\u002Fconfigs\u002Fexecution\u002Finternal\u002Fslurm\u002F*.yaml; do \\\n  echo \"$(basename \"$f\" .yaml) -> $(grep -E '^hostname:' \"$f\" | awk '{print $2}')\"; done\n",[4716],{"type":40,"tag":67,"props":4717,"children":4718},{"__ignoreMap":103},[4719,4765,4811,4861],{"type":40,"tag":2054,"props":4720,"children":4721},{"class":2056,"line":2057},[4722,4727,4731,4736,4741,4745,4750,4755,4760],{"type":40,"tag":2054,"props":4723,"children":4724},{"style":2061},[4725],{"type":46,"value":4726},"python3",{"type":40,"tag":2054,"props":4728,"children":4729},{"style":2066},[4730],{"type":46,"value":2457},{"type":40,"tag":2054,"props":4732,"children":4733},{"style":2082},[4734],{"type":46,"value":4735}," '",{"type":40,"tag":2054,"props":4737,"children":4738},{"style":2066},[4739],{"type":46,"value":4740},"import nemo_evaluator_launcher_internal",{"type":40,"tag":2054,"props":4742,"children":4743},{"style":2082},[4744],{"type":46,"value":2503},{"type":40,"tag":2054,"props":4746,"children":4747},{"style":2082},[4748],{"type":46,"value":4749}," 2>",{"type":40,"tag":2054,"props":4751,"children":4752},{"style":2066},[4753],{"type":46,"value":4754},"\u002Fdev\u002Fnull",{"type":40,"tag":2054,"props":4756,"children":4757},{"style":2082},[4758],{"type":46,"value":4759}," &&",{"type":40,"tag":2054,"props":4761,"children":4762},{"style":2093},[4763],{"type":46,"value":4764}," \\\n",{"type":40,"tag":2054,"props":4766,"children":4767},{"class":2056,"line":2465},[4768,4773,4778,4782,4786,4790,4795,4799,4803,4807],{"type":40,"tag":2054,"props":4769,"children":4770},{"style":2093},[4771],{"type":46,"value":4772},"PKG",{"type":40,"tag":2054,"props":4774,"children":4775},{"style":2082},[4776],{"type":46,"value":4777},"=$(",{"type":40,"tag":2054,"props":4779,"children":4780},{"style":2061},[4781],{"type":46,"value":4726},{"type":40,"tag":2054,"props":4783,"children":4784},{"style":2066},[4785],{"type":46,"value":2457},{"type":40,"tag":2054,"props":4787,"children":4788},{"style":2082},[4789],{"type":46,"value":4735},{"type":40,"tag":2054,"props":4791,"children":4792},{"style":2066},[4793],{"type":46,"value":4794},"import nemo_evaluator_launcher_internal as m,os;print(os.path.dirname(m.__file__))",{"type":40,"tag":2054,"props":4796,"children":4797},{"style":2082},[4798],{"type":46,"value":2503},{"type":40,"tag":2054,"props":4800,"children":4801},{"style":2082},[4802],{"type":46,"value":3172},{"type":40,"tag":2054,"props":4804,"children":4805},{"style":2082},[4806],{"type":46,"value":4759},{"type":40,"tag":2054,"props":4808,"children":4809},{"style":2093},[4810],{"type":46,"value":4764},{"type":40,"tag":2054,"props":4812,"children":4813},{"class":2056,"line":2474},[4814,4819,4824,4829,4833,4838,4842,4847,4852,4857],{"type":40,"tag":2054,"props":4815,"children":4816},{"style":3342},[4817],{"type":46,"value":4818},"for",{"type":40,"tag":2054,"props":4820,"children":4821},{"style":2093},[4822],{"type":46,"value":4823}," f ",{"type":40,"tag":2054,"props":4825,"children":4826},{"style":3342},[4827],{"type":46,"value":4828},"in",{"type":40,"tag":2054,"props":4830,"children":4831},{"style":2082},[4832],{"type":46,"value":2417},{"type":40,"tag":2054,"props":4834,"children":4835},{"style":2093},[4836],{"type":46,"value":4837},"$PKG",{"type":40,"tag":2054,"props":4839,"children":4840},{"style":2082},[4841],{"type":46,"value":2427},{"type":40,"tag":2054,"props":4843,"children":4844},{"style":2066},[4845],{"type":46,"value":4846},"\u002Fconfigs\u002Fexecution\u002Finternal\u002Fslurm\u002F*.yaml",{"type":40,"tag":2054,"props":4848,"children":4849},{"style":2082},[4850],{"type":46,"value":4851},";",{"type":40,"tag":2054,"props":4853,"children":4854},{"style":3342},[4855],{"type":46,"value":4856}," do",{"type":40,"tag":2054,"props":4858,"children":4859},{"style":2093},[4860],{"type":46,"value":4764},{"type":40,"tag":2054,"props":4862,"children":4863},{"class":2056,"line":2483},[4864,4870,4875,4879,4883,4888,4892,4897,4901,4906,4911,4916,4921,4925,4930,4934,4938,4942,4946,4951,4956,4960,4965,4970,4974],{"type":40,"tag":2054,"props":4865,"children":4867},{"style":4866},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[4868],{"type":46,"value":4869},"  echo",{"type":40,"tag":2054,"props":4871,"children":4872},{"style":2082},[4873],{"type":46,"value":4874}," \"$(",{"type":40,"tag":2054,"props":4876,"children":4877},{"style":2061},[4878],{"type":46,"value":2549},{"type":40,"tag":2054,"props":4880,"children":4881},{"style":2082},[4882],{"type":46,"value":2417},{"type":40,"tag":2054,"props":4884,"children":4885},{"style":2093},[4886],{"type":46,"value":4887},"$f",{"type":40,"tag":2054,"props":4889,"children":4890},{"style":2082},[4891],{"type":46,"value":2427},{"type":40,"tag":2054,"props":4893,"children":4894},{"style":2066},[4895],{"type":46,"value":4896}," .yaml",{"type":40,"tag":2054,"props":4898,"children":4899},{"style":2082},[4900],{"type":46,"value":3172},{"type":40,"tag":2054,"props":4902,"children":4903},{"style":2066},[4904],{"type":46,"value":4905}," -> ",{"type":40,"tag":2054,"props":4907,"children":4908},{"style":2082},[4909],{"type":46,"value":4910},"$(",{"type":40,"tag":2054,"props":4912,"children":4913},{"style":2061},[4914],{"type":46,"value":4915},"grep",{"type":40,"tag":2054,"props":4917,"children":4918},{"style":2066},[4919],{"type":46,"value":4920}," -E ",{"type":40,"tag":2054,"props":4922,"children":4923},{"style":2082},[4924],{"type":46,"value":2503},{"type":40,"tag":2054,"props":4926,"children":4927},{"style":2066},[4928],{"type":46,"value":4929},"^hostname:",{"type":40,"tag":2054,"props":4931,"children":4932},{"style":2082},[4933],{"type":46,"value":2503},{"type":40,"tag":2054,"props":4935,"children":4936},{"style":2082},[4937],{"type":46,"value":2417},{"type":40,"tag":2054,"props":4939,"children":4940},{"style":2093},[4941],{"type":46,"value":4887},{"type":40,"tag":2054,"props":4943,"children":4944},{"style":2082},[4945],{"type":46,"value":2427},{"type":40,"tag":2054,"props":4947,"children":4948},{"style":2082},[4949],{"type":46,"value":4950}," |",{"type":40,"tag":2054,"props":4952,"children":4953},{"style":2061},[4954],{"type":46,"value":4955}," awk",{"type":40,"tag":2054,"props":4957,"children":4958},{"style":2082},[4959],{"type":46,"value":4735},{"type":40,"tag":2054,"props":4961,"children":4962},{"style":2066},[4963],{"type":46,"value":4964},"{print $2}",{"type":40,"tag":2054,"props":4966,"children":4967},{"style":2082},[4968],{"type":46,"value":4969},"')\"",{"type":40,"tag":2054,"props":4971,"children":4972},{"style":2082},[4973],{"type":46,"value":4851},{"type":40,"tag":2054,"props":4975,"children":4976},{"style":3342},[4977],{"type":46,"value":4978}," done\n",{"type":40,"tag":49,"props":4980,"children":4981},{},[4982,4984,4990,4992,4998,5000,5006,5008,5014],{"type":46,"value":4983},"Hostname match → set ",{"type":40,"tag":67,"props":4985,"children":4987},{"className":4986},[],[4988],{"type":46,"value":4989},"defaults: - execution: internal\u002Fslurm\u002F\u003Ccluster>",{"type":46,"value":4991},", drop the redundant ",{"type":40,"tag":67,"props":4993,"children":4995},{"className":4994},[],[4996],{"type":46,"value":4997},"execution.hostname",{"type":46,"value":4999}," (keep account\u002Foutput_dir\u002Fwalltime), verify with ",{"type":40,"tag":67,"props":5001,"children":5003},{"className":5002},[],[5004],{"type":46,"value":5005},"--dry-run",{"type":46,"value":5007},". Else keep ",{"type":40,"tag":67,"props":5009,"children":5011},{"className":5010},[],[5012],{"type":46,"value":5013},"slurm\u002Fdefault",{"type":46,"value":5015}," and fill hostname\u002Faccount\u002Foutput_dir manually.",{"type":40,"tag":49,"props":5017,"children":5018},{},[5019,5021,5026,5028,5034,5036,5042,5044,5050,5052,5058],{"type":46,"value":5020},"On SLURM, several deploy\u002Feval failures are invisible to ",{"type":40,"tag":67,"props":5022,"children":5024},{"className":5023},[],[5025],{"type":46,"value":5005},{"type":46,"value":5027}," and only surface at canary (",{"type":40,"tag":67,"props":5029,"children":5031},{"className":5030},[],[5032],{"type":46,"value":5033},"mount_home",{"type":46,"value":5035},", HF cache, ",{"type":40,"tag":67,"props":5037,"children":5039},{"className":5038},[],[5040],{"type":46,"value":5041},"cpu_partition",{"type":46,"value":5043},", top-level vs per-stage ",{"type":40,"tag":67,"props":5045,"children":5047},{"className":5046},[],[5048],{"type":46,"value":5049},"env_vars",{"type":46,"value":5051},") — read ",{"type":40,"tag":67,"props":5053,"children":5055},{"className":5054},[],[5056],{"type":46,"value":5057},"references\u002Fslurm.md",{"type":46,"value":258},{"type":40,"tag":1329,"props":5060,"children":5061},{},[5062,5106,5142,5147],{"type":40,"tag":227,"props":5063,"children":5064},{},[5065,5067,5072,5074,5079,5081,5086,5087,5091,5093,5098,5099,5104],{"type":46,"value":5066},"Find every ",{"type":40,"tag":67,"props":5068,"children":5070},{"className":5069},[],[5071],{"type":46,"value":874},{"type":46,"value":5073}," left. Ask the user only for what can't be inferred (SLURM hostname\u002Faccount\u002Foutput_dir, the ",{"type":40,"tag":67,"props":5075,"children":5077},{"className":5076},[],[5078],{"type":46,"value":5041},{"type":46,"value":5080}," for auto-export, etc.). Don't propose defaults; let them give plain text. (",{"type":40,"tag":67,"props":5082,"children":5084},{"className":5083},[],[5085],{"type":46,"value":1912},{"type":46,"value":868},{"type":40,"tag":125,"props":5088,"children":5089},{},[5090],{"type":46,"value":136},{"type":46,"value":5092}," one of these — it's ",{"type":40,"tag":67,"props":5094,"children":5096},{"className":5095},[],[5097],{"type":46,"value":1867},{"type":46,"value":1921},{"type":40,"tag":67,"props":5100,"children":5102},{"className":5101},[],[5103],{"type":46,"value":293},{"type":46,"value":5105},".)",{"type":40,"tag":227,"props":5107,"children":5108},{},[5109,5117,5119,5125,5127,5133,5135,5140],{"type":40,"tag":125,"props":5110,"children":5111},{},[5112],{"type":40,"tag":67,"props":5113,"children":5115},{"className":5114},[],[5116],{"type":46,"value":3903},{"type":46,"value":5118}," — size it yourself from the run shape (total requests = ",{"type":40,"tag":67,"props":5120,"children":5122},{"className":5121},[],[5123],{"type":46,"value":5124},"dataset_size × repeats",{"type":46,"value":5126}," vs GPU serving capacity), and set ",{"type":40,"tag":67,"props":5128,"children":5130},{"className":5129},[],[5131],{"type":46,"value":5132},"--max-num-seqs",{"type":46,"value":5134}," to match. Read ",{"type":40,"tag":67,"props":5136,"children":5138},{"className":5137},[],[5139],{"type":46,"value":3506},{"type":46,"value":5141}," for the decision rule and worked examples; only ask the user if a non-GPU cap (e.g. judge rate limit) is unknown.",{"type":40,"tag":227,"props":5143,"children":5144},{},[5145],{"type":46,"value":5146},"Ask about other defaults they may want to change (partition, walltime, MLflow tags).",{"type":40,"tag":227,"props":5148,"children":5149},{},[5150,5159,5161,5166,5168,5173,5175,5181,5183,5188,5189,5194,5196,5202,5204,5209,5211,5217,5219,5225],{"type":40,"tag":125,"props":5151,"children":5152},{},[5153],{"type":40,"tag":67,"props":5154,"children":5156},{"className":5155},[],[5157],{"type":46,"value":5158},"execution.gres",{"type":46,"value":5160}," — auto-set if you used a predefined ",{"type":40,"tag":67,"props":5162,"children":5164},{"className":5163},[],[5165],{"type":46,"value":4701},{"type":46,"value":5167}," config (above). On the ",{"type":40,"tag":67,"props":5169,"children":5171},{"className":5170},[],[5172],{"type":46,"value":5013},{"type":46,"value":5174}," fallback it's ",{"type":40,"tag":67,"props":5176,"children":5178},{"className":5177},[],[5179],{"type":46,"value":5180},"gpu:8",{"type":46,"value":5182},", so set it to the node's GPU count (and match ",{"type":40,"tag":67,"props":5184,"children":5186},{"className":5185},[],[5187],{"type":46,"value":3273},{"type":46,"value":185},{"type":40,"tag":67,"props":5190,"children":5192},{"className":5191},[],[5193],{"type":46,"value":3266},{"type":46,"value":5195},") or ",{"type":40,"tag":67,"props":5197,"children":5199},{"className":5198},[],[5200],{"type":46,"value":5201},"sbatch",{"type":46,"value":5203}," rejects the job with ",{"type":40,"tag":761,"props":5205,"children":5206},{},[5207],{"type":46,"value":5208},"\"Requested node configuration is not available\"",{"type":46,"value":5210}," (e.g. 4-GPU GB300 → ",{"type":40,"tag":67,"props":5212,"children":5214},{"className":5213},[],[5215],{"type":46,"value":5216},"gres: gpu:4",{"type":46,"value":5218},"; check with ",{"type":40,"tag":67,"props":5220,"children":5222},{"className":5221},[],[5223],{"type":46,"value":5224},"sinfo -o '%P %G'",{"type":46,"value":90},{"type":40,"tag":49,"props":5227,"children":5228},{},[5229,5234,5236,5242],{"type":40,"tag":125,"props":5230,"children":5231},{},[5232],{"type":46,"value":5233},"Walltime cap: 4 hours.",{"type":46,"value":5235}," Always ",{"type":40,"tag":67,"props":5237,"children":5239},{"className":5238},[],[5240],{"type":46,"value":5241},"execution.walltime: \"04:00:00\"",{"type":46,"value":5243},". The cluster does not schedule jobs longer than 4h — this is a hard limit, not a preference.",{"type":40,"tag":49,"props":5245,"children":5246},{},[5247,5249,5254,5256,5262],{"type":46,"value":5248},"Evals that exceed 4h of wall-clock time are handled by ",{"type":40,"tag":125,"props":5250,"children":5251},{},[5252],{"type":46,"value":5253},"NEL's built-in dependency-chain resume",{"type":46,"value":5255},", not by shrinking the eval. NEL submits the first SLURM job; if it hits walltime, a dependent follow-on job resumes from the response\u002Fresult caches the first job wrote, then queues another follow-on. Long evals continue across walltime windows automatically. See ",{"type":40,"tag":67,"props":5257,"children":5259},{"className":5258},[],[5260],{"type":46,"value":5261},"references\u002Frun-validation.md#nel-timeout-and-resume-behavior",{"type":46,"value":5263}," for the full mechanism.",{"type":40,"tag":49,"props":5265,"children":5266},{},[5267,5280,5282,5288,5290,5296,5298,5303],{"type":40,"tag":125,"props":5268,"children":5269},{},[5270,5272,5278],{"type":46,"value":5271},"Never ",{"type":40,"tag":67,"props":5273,"children":5275},{"className":5274},[],[5276],{"type":46,"value":5277},"scancel",{"type":46,"value":5279}," a wedged run to free the GPUs.",{"type":46,"value":5281}," The dependency-chain resume above fires only on a genuine walltime timeout; ",{"type":40,"tag":67,"props":5283,"children":5285},{"className":5284},[],[5286],{"type":46,"value":5287},"afternotok",{"type":46,"value":5289}," explicitly refuses a predecessor that finished ",{"type":40,"tag":67,"props":5291,"children":5293},{"className":5292},[],[5294],{"type":46,"value":5295},"CANCELLED by \u003Cuid>",{"type":46,"value":5297}," and exits in seconds, discarding the response cache the run had already filled. Let it hit the wall clock instead — one ",{"type":40,"tag":67,"props":5299,"children":5301},{"className":5300},[],[5302],{"type":46,"value":5277},{"type":46,"value":5304}," threw away ~59 GPU-h of completed generation.",{"type":40,"tag":49,"props":5306,"children":5307},{},[5308,5320,5322,5327,5329,5335,5337,5342,5344,5350,5352,5358,5360,5364,5366,5372,5374,5379,5381,5387,5389,5395],{"type":40,"tag":125,"props":5309,"children":5310},{},[5311,5313,5319],{"type":46,"value":5312},"Preemption \u002F external kill — resume manually with ",{"type":40,"tag":67,"props":5314,"children":5316},{"className":5315},[],[5317],{"type":46,"value":5318},"sbatch run.sub",{"type":46,"value":258},{"type":46,"value":5321}," On a preemptible account (common on busy internal clusters) the scheduler can ",{"type":40,"tag":125,"props":5323,"children":5324},{},[5325],{"type":46,"value":5326},"CANCEL",{"type":46,"value":5328}," a run mid-eval for a higher-priority job — ",{"type":40,"tag":67,"props":5330,"children":5332},{"className":5331},[],[5333],{"type":46,"value":5334},"sacct -j \u003Cid>",{"type":46,"value":5336}," shows ",{"type":40,"tag":67,"props":5338,"children":5340},{"className":5339},[],[5341],{"type":46,"value":5295},{"type":46,"value":5343}," (a ",{"type":40,"tag":67,"props":5345,"children":5347},{"className":5346},[],[5348],{"type":46,"value":5349},"svc-*",{"type":46,"value":5351}," service account) with ",{"type":40,"tag":67,"props":5353,"children":5355},{"className":5354},[],[5356],{"type":46,"value":5357},"Elapsed",{"type":46,"value":5359}," well under the 4h walltime. NEL does ",{"type":40,"tag":125,"props":5361,"children":5362},{},[5363],{"type":46,"value":136},{"type":46,"value":5365}," auto-resume this (its dependency chain only fires on a genuine walltime timeout). But the ",{"type":40,"tag":67,"props":5367,"children":5369},{"className":5368},[],[5370],{"type":46,"value":5371},"run.sub",{"type":46,"value":5373}," that NEL generated for the job (in its run dir) is ",{"type":40,"tag":125,"props":5375,"children":5376},{},[5377],{"type":46,"value":5378},"re-submittable",{"type":46,"value":5380}," and resumes from the same ",{"type":40,"tag":67,"props":5382,"children":5384},{"className":5383},[],[5385],{"type":46,"value":5386},"output_dir",{"type":46,"value":5388}," + response cache (",{"type":40,"tag":67,"props":5390,"children":5392},{"className":5391},[],[5393],{"type":46,"value":5394},"skip_filled",{"type":46,"value":5396},"), continuing from the partial output rather than restarting:",{"type":40,"tag":98,"props":5398,"children":5400},{"className":2046,"code":5399,"language":2048,"meta":103,"style":103},"ssh \u003Chost> \"cd \u003Coutput_dir>\u002F\u003Ctimestamp>-\u003Cinvocation>\u002F\u003Ctask>\u002F && sbatch run.sub\"\n",[5401],{"type":40,"tag":67,"props":5402,"children":5403},{"__ignoreMap":103},[5404],{"type":40,"tag":2054,"props":5405,"children":5406},{"class":2056,"line":2057},[5407,5412,5416,5421,5426,5430,5434,5439],{"type":40,"tag":2054,"props":5408,"children":5409},{"style":2061},[5410],{"type":46,"value":5411},"ssh",{"type":40,"tag":2054,"props":5413,"children":5414},{"style":2082},[5415],{"type":46,"value":2085},{"type":40,"tag":2054,"props":5417,"children":5418},{"style":2066},[5419],{"type":46,"value":5420},"hos",{"type":40,"tag":2054,"props":5422,"children":5423},{"style":2093},[5424],{"type":46,"value":5425},"t",{"type":40,"tag":2054,"props":5427,"children":5428},{"style":2082},[5429],{"type":46,"value":2100},{"type":40,"tag":2054,"props":5431,"children":5432},{"style":2082},[5433],{"type":46,"value":2417},{"type":40,"tag":2054,"props":5435,"children":5436},{"style":2066},[5437],{"type":46,"value":5438},"cd \u003Coutput_dir>\u002F\u003Ctimestamp>-\u003Cinvocation>\u002F\u003Ctask>\u002F && sbatch run.sub",{"type":40,"tag":2054,"props":5440,"children":5441},{"style":2082},[5442],{"type":46,"value":5443},"\"\n",{"type":40,"tag":49,"props":5445,"children":5446},{},[5447,5449,5454,5456,5461,5463,5469],{"type":46,"value":5448},"Re-submit again if it's preempted again — each resume re-deploys, then skips already-generated samples, so progress is ",{"type":40,"tag":125,"props":5450,"children":5451},{},[5452],{"type":46,"value":5453},"cumulative",{"type":46,"value":5455}," across attempts until it completes. Always confirm via ",{"type":40,"tag":67,"props":5457,"children":5459},{"className":5458},[],[5460],{"type":46,"value":5334},{"type":46,"value":5462}," that the prior job was ",{"type":40,"tag":67,"props":5464,"children":5466},{"className":5465},[],[5467],{"type":46,"value":5468},"CANCELLED",{"type":46,"value":5470}," (not a real failure) before resuming.",{"type":40,"tag":49,"props":5472,"children":5473},{},[5474],{"type":46,"value":5475},"Implications for the agent:",{"type":40,"tag":1329,"props":5477,"children":5478},{},[5479,5498,5530],{"type":40,"tag":227,"props":5480,"children":5481},{},[5482,5484,5488,5490,5496],{"type":46,"value":5483},"Do ",{"type":40,"tag":125,"props":5485,"children":5486},{},[5487],{"type":46,"value":136},{"type":46,"value":5489}," lower ",{"type":40,"tag":67,"props":5491,"children":5493},{"className":5492},[],[5494],{"type":46,"value":5495},"num_repeats",{"type":46,"value":5497},", split heavy tasks (AA-LCR, SciCode) into separate configs, or otherwise carve up the eval to fit inside 4h. Let NEL chain.",{"type":40,"tag":227,"props":5499,"children":5500},{},[5501,5502,5506,5508,5514,5515,5521,5523,5528],{"type":46,"value":5483},{"type":40,"tag":125,"props":5503,"children":5504},{},[5505],{"type":46,"value":136},{"type":46,"value":5507}," treat a walltime timeout as a failed run. Check ",{"type":40,"tag":67,"props":5509,"children":5511},{"className":5510},[],[5512],{"type":46,"value":5513},"nel status",{"type":46,"value":744},{"type":40,"tag":67,"props":5516,"children":5518},{"className":5517},[],[5519],{"type":46,"value":5520},"nel info",{"type":46,"value":5522}," and the dependent job's logs before declaring failure. ",{"type":40,"tag":67,"props":5524,"children":5526},{"className":5525},[],[5527],{"type":46,"value":4402},{"type":46,"value":5529}," covers what a real failure looks like vs an expected resume event.",{"type":40,"tag":227,"props":5531,"children":5532},{},[5533,5535,5540,5541,5546],{"type":46,"value":5534},"Bumping ",{"type":40,"tag":67,"props":5536,"children":5538},{"className":5537},[],[5539],{"type":46,"value":3295},{"type":46,"value":744},{"type":40,"tag":67,"props":5542,"children":5544},{"className":5543},[],[5545],{"type":46,"value":3903},{"type":46,"value":5547}," to finish faster is fine when the goal is wall-clock latency, not a walltime workaround — but it's optional, not required, for runs longer than 4h.",{"type":40,"tag":110,"props":5549,"children":5550},{},[],{"type":40,"tag":55,"props":5552,"children":5554},{"id":5553},"step-5-confirm-tasks-iterative",[5555],{"type":46,"value":5556},"Step 5 — Confirm tasks (iterative)",{"type":40,"tag":223,"props":5558,"children":5559},{},[5560,5573,5586,5698],{"type":40,"tag":227,"props":5561,"children":5562},{},[5563,5565,5571],{"type":46,"value":5564},"Tell user: \"Run ",{"type":40,"tag":67,"props":5566,"children":5568},{"className":5567},[],[5569],{"type":46,"value":5570},"nel ls tasks",{"type":46,"value":5572}," for the full task list.\"",{"type":40,"tag":227,"props":5574,"children":5575},{},[5576,5578,5584],{"type":46,"value":5577},"For any task with a ",{"type":40,"tag":67,"props":5579,"children":5581},{"className":5580},[],[5582],{"type":46,"value":5583},"recipes\u002Ftasks\u002F",{"type":46,"value":5585}," reference, read it and prefer its YAML fragment + repeat counts.",{"type":40,"tag":227,"props":5587,"children":5588},{},[5589,5591,5596,5597],{"type":46,"value":5590},"Ask about add\u002Fremove\u002Fmodify. Per-task overrides under task's ",{"type":40,"tag":67,"props":5592,"children":5594},{"className":5593},[],[5595],{"type":46,"value":3970},{"type":46,"value":2829},{"type":40,"tag":98,"props":5598,"children":5600},{"className":3307,"code":5599,"language":3309,"meta":103,"style":103},"tasks:\n  - name: \u003Ctask>\n    nemo_evaluator_config:\n      config:\n        params:\n          temperature: \u003Cvalue>\n          ...\n",[5601],{"type":40,"tag":67,"props":5602,"children":5603},{"__ignoreMap":103},[5604,5615,5637,5649,5661,5673,5690],{"type":40,"tag":2054,"props":5605,"children":5606},{"class":2056,"line":2057},[5607,5611],{"type":40,"tag":2054,"props":5608,"children":5609},{"style":3319},[5610],{"type":46,"value":1513},{"type":40,"tag":2054,"props":5612,"children":5613},{"style":2082},[5614],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5616,"children":5617},{"class":2056,"line":2465},[5618,5623,5628,5632],{"type":40,"tag":2054,"props":5619,"children":5620},{"style":2082},[5621],{"type":46,"value":5622},"  -",{"type":40,"tag":2054,"props":5624,"children":5625},{"style":3319},[5626],{"type":46,"value":5627}," name",{"type":40,"tag":2054,"props":5629,"children":5630},{"style":2082},[5631],{"type":46,"value":2829},{"type":40,"tag":2054,"props":5633,"children":5634},{"style":2066},[5635],{"type":46,"value":5636}," \u003Ctask>\n",{"type":40,"tag":2054,"props":5638,"children":5639},{"class":2056,"line":2474},[5640,5645],{"type":40,"tag":2054,"props":5641,"children":5642},{"style":3319},[5643],{"type":46,"value":5644},"    nemo_evaluator_config",{"type":40,"tag":2054,"props":5646,"children":5647},{"style":2082},[5648],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5650,"children":5651},{"class":2056,"line":2483},[5652,5657],{"type":40,"tag":2054,"props":5653,"children":5654},{"style":3319},[5655],{"type":46,"value":5656},"      config",{"type":40,"tag":2054,"props":5658,"children":5659},{"style":2082},[5660],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5662,"children":5663},{"class":2056,"line":2492},[5664,5669],{"type":40,"tag":2054,"props":5665,"children":5666},{"style":3319},[5667],{"type":46,"value":5668},"        params",{"type":40,"tag":2054,"props":5670,"children":5671},{"style":2082},[5672],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5674,"children":5675},{"class":2056,"line":3378},[5676,5681,5685],{"type":40,"tag":2054,"props":5677,"children":5678},{"style":3319},[5679],{"type":46,"value":5680},"          temperature",{"type":40,"tag":2054,"props":5682,"children":5683},{"style":2082},[5684],{"type":46,"value":2829},{"type":40,"tag":2054,"props":5686,"children":5687},{"style":2066},[5688],{"type":46,"value":5689}," \u003Cvalue>\n",{"type":40,"tag":2054,"props":5691,"children":5692},{"class":2056,"line":3387},[5693],{"type":40,"tag":2054,"props":5694,"children":5695},{"style":4088},[5696],{"type":46,"value":5697},"          ...\n",{"type":40,"tag":227,"props":5699,"children":5700},{},[5701],{"type":46,"value":5702},"Apply, show updated list, ask \"Final, or more changes?\" Loop until confirmed.",{"type":40,"tag":49,"props":5704,"children":5705},{},[5706,5711,5713,5718,5719,5724,5726,5731,5733,5737,5739,5744,5746,5752,5754,5759,5761,5765,5767,5773,5775,5781,5783,5789,5791,5796,5798,5803,5805,5810,5812,5818,5819,5825,5827,5832,5834,5839],{"type":40,"tag":125,"props":5707,"children":5708},{},[5709],{"type":46,"value":5710},"Tasks that call an external judge \u002F user-simulator \u002F scoring endpoint.",{"type":46,"value":5712}," Treat this as a general pattern, not a fixed list — HLE, AA-LCR, and Tau2 need one today, but other benchmarks may too (check each task's recipe). Their ",{"type":40,"tag":67,"props":5714,"children":5716},{"className":5715},[],[5717],{"type":46,"value":1240},{"type":46,"value":744},{"type":40,"tag":67,"props":5720,"children":5722},{"className":5721},[],[5723],{"type":46,"value":1247},{"type":46,"value":5725}," are ",{"type":40,"tag":125,"props":5727,"children":5728},{},[5729],{"type":46,"value":5730},"config, not secrets",{"type":46,"value":5732},": substitute the ",{"type":40,"tag":125,"props":5734,"children":5735},{},[5736],{"type":46,"value":1794},{"type":46,"value":5738}," values the user keeps in ",{"type":40,"tag":67,"props":5740,"children":5742},{"className":5741},[],[5743],{"type":46,"value":317},{"type":46,"value":5745}," (keys per the task's recipe + ",{"type":40,"tag":67,"props":5747,"children":5749},{"className":5748},[],[5750],{"type":46,"value":5751},"recipes\u002Fenv.example",{"type":46,"value":5753},") into the task's ",{"type":40,"tag":67,"props":5755,"children":5757},{"className":5756},[],[5758],{"type":46,"value":1255},{"type":46,"value":5760}," placeholders. Do ",{"type":40,"tag":125,"props":5762,"children":5763},{},[5764],{"type":46,"value":136},{"type":46,"value":5766}," emit ",{"type":40,"tag":67,"props":5768,"children":5770},{"className":5769},[],[5771],{"type":46,"value":5772},"${oc.env:...}",{"type":46,"value":5774}," for these (it silently fails unless the var was exported with ",{"type":40,"tag":67,"props":5776,"children":5778},{"className":5777},[],[5779],{"type":46,"value":5780},"set -a",{"type":46,"value":5782},"). Only ",{"type":40,"tag":67,"props":5784,"children":5786},{"className":5785},[],[5787],{"type":46,"value":5788},"api_key",{"type":46,"value":5790}," stays an env-var ",{"type":40,"tag":761,"props":5792,"children":5793},{},[5794],{"type":46,"value":5795},"name",{"type":46,"value":5797}," (e.g. ",{"type":40,"tag":67,"props":5799,"children":5801},{"className":5800},[],[5802],{"type":46,"value":555},{"type":46,"value":5804},"), exported and read by the harness. All judges + user-sims (HLE, AA-LCR, Tau2, AIME) use one ",{"type":40,"tag":67,"props":5806,"children":5808},{"className":5807},[],[5809],{"type":46,"value":555},{"type":46,"value":5811}," against one OpenAI-compatible host (AIME's simple-evals default judge endpoint is overridden to it — see its recipe). Get the ",{"type":40,"tag":67,"props":5813,"children":5815},{"className":5814},[],[5816],{"type":46,"value":5817},"*_MODEL_ID",{"type":46,"value":185},{"type":40,"tag":67,"props":5820,"children":5822},{"className":5821},[],[5823],{"type":46,"value":5824},"*_URL",{"type":46,"value":5826}," values via ",{"type":40,"tag":67,"props":5828,"children":5830},{"className":5829},[],[5831],{"type":46,"value":293},{"type":46,"value":5833}," (see Step 1) rather than guessing a host; only fill them by hand if it's unavailable. ",{"type":40,"tag":67,"props":5835,"children":5837},{"className":5836},[],[5838],{"type":46,"value":317},{"type":46,"value":5840}," should already exist (Step 1) — if not, set it up now (don't defer to Step 8) before substituting.",{"type":40,"tag":49,"props":5842,"children":5843},{},[5844,5849,5851,5857,5859,5865,5867,5871],{"type":40,"tag":125,"props":5845,"children":5846},{},[5847],{"type":46,"value":5848},"Known issue — nemo-skills self-deployment:",{"type":46,"value":5850}," If using ",{"type":40,"tag":67,"props":5852,"children":5854},{"className":5853},[],[5855],{"type":46,"value":5856},"nemo_skills.*",{"type":46,"value":5858}," tasks (",{"type":40,"tag":67,"props":5860,"children":5862},{"className":5861},[],[5863],{"type":46,"value":5864},"ns_*",{"type":46,"value":5866},") with self-deployment (vLLM\u002FSGLang\u002FNIM), you need ",{"type":40,"tag":125,"props":5868,"children":5869},{},[5870],{"type":46,"value":772},{"type":46,"value":5872}," of these:",{"type":40,"tag":98,"props":5874,"children":5876},{"className":3307,"code":5875,"language":3309,"meta":103,"style":103},"evaluation:\n  env_vars:\n    DUMMY_API_KEY: lit:dummy   # MUST be set here — see below\n  nemo_evaluator_config:\n    target:\n      api_endpoint:\n        api_key_name: DUMMY_API_KEY\n",[5877],{"type":40,"tag":67,"props":5878,"children":5879},{"__ignoreMap":103},[5880,5891,5903,5925,5937,5949,5961],{"type":40,"tag":2054,"props":5881,"children":5882},{"class":2056,"line":2057},[5883,5887],{"type":40,"tag":2054,"props":5884,"children":5885},{"style":3319},[5886],{"type":46,"value":4},{"type":40,"tag":2054,"props":5888,"children":5889},{"style":2082},[5890],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5892,"children":5893},{"class":2056,"line":2465},[5894,5899],{"type":40,"tag":2054,"props":5895,"children":5896},{"style":3319},[5897],{"type":46,"value":5898},"  env_vars",{"type":40,"tag":2054,"props":5900,"children":5901},{"style":2082},[5902],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5904,"children":5905},{"class":2056,"line":2474},[5906,5911,5915,5920],{"type":40,"tag":2054,"props":5907,"children":5908},{"style":3319},[5909],{"type":46,"value":5910},"    DUMMY_API_KEY",{"type":40,"tag":2054,"props":5912,"children":5913},{"style":2082},[5914],{"type":46,"value":2829},{"type":40,"tag":2054,"props":5916,"children":5917},{"style":2066},[5918],{"type":46,"value":5919}," lit:dummy",{"type":40,"tag":2054,"props":5921,"children":5922},{"style":4070},[5923],{"type":46,"value":5924},"   # MUST be set here — see below\n",{"type":40,"tag":2054,"props":5926,"children":5927},{"class":2056,"line":2483},[5928,5933],{"type":40,"tag":2054,"props":5929,"children":5930},{"style":3319},[5931],{"type":46,"value":5932},"  nemo_evaluator_config",{"type":40,"tag":2054,"props":5934,"children":5935},{"style":2082},[5936],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5938,"children":5939},{"class":2056,"line":2492},[5940,5945],{"type":40,"tag":2054,"props":5941,"children":5942},{"style":3319},[5943],{"type":46,"value":5944},"    target",{"type":40,"tag":2054,"props":5946,"children":5947},{"style":2082},[5948],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5950,"children":5951},{"class":2056,"line":3378},[5952,5957],{"type":40,"tag":2054,"props":5953,"children":5954},{"style":3319},[5955],{"type":46,"value":5956},"      api_endpoint",{"type":40,"tag":2054,"props":5958,"children":5959},{"style":2082},[5960],{"type":46,"value":3327},{"type":40,"tag":2054,"props":5962,"children":5963},{"class":2056,"line":3387},[5964,5969,5973],{"type":40,"tag":2054,"props":5965,"children":5966},{"style":3319},[5967],{"type":46,"value":5968},"        api_key_name",{"type":40,"tag":2054,"props":5970,"children":5971},{"style":2082},[5972],{"type":46,"value":2829},{"type":40,"tag":2054,"props":5974,"children":5975},{"style":2066},[5976],{"type":46,"value":5977}," DUMMY_API_KEY\n",{"type":40,"tag":49,"props":5979,"children":5980},{},[5981,5987,5989,5994,5995,6001,6003,6009,6011,6016,6018,6023,6025,6031,6033,6039,6041,6046,6048,6053],{"type":40,"tag":67,"props":5982,"children":5984},{"className":5983},[],[5985],{"type":46,"value":5986},"api_key_name",{"type":46,"value":5988}," only names the env var; the nemo-skills client ",{"type":40,"tag":125,"props":5990,"children":5991},{},[5992],{"type":46,"value":5993},"hard-fails if that var has no value inside the eval container",{"type":46,"value":736},{"type":40,"tag":67,"props":5996,"children":5998},{"className":5997},[],[5999],{"type":46,"value":6000},"ValueError: api_key_env_var=DUMMY_API_KEY but the value is not set",{"type":46,"value":6002},"). On SLURM, a shell ",{"type":40,"tag":67,"props":6004,"children":6006},{"className":6005},[],[6007],{"type":46,"value":6008},"export DUMMY_API_KEY=dummy",{"type":46,"value":6010}," (Step 8) does ",{"type":40,"tag":125,"props":6012,"children":6013},{},[6014],{"type":46,"value":6015},"NOT",{"type":46,"value":6017}," propagate into the container — NEL only injects vars declared in ",{"type":40,"tag":67,"props":6019,"children":6021},{"className":6020},[],[6022],{"type":46,"value":5049},{"type":46,"value":6024},". So declare ",{"type":40,"tag":67,"props":6026,"children":6028},{"className":6027},[],[6029],{"type":46,"value":6030},"DUMMY_API_KEY: lit:dummy",{"type":46,"value":6032}," under ",{"type":40,"tag":67,"props":6034,"children":6036},{"className":6035},[],[6037],{"type":46,"value":6038},"evaluation.env_vars",{"type":46,"value":6040}," (note the ",{"type":40,"tag":67,"props":6042,"children":6044},{"className":6043},[],[6045],{"type":46,"value":2797},{"type":46,"value":6047}," prefix — see below). The shell export only helps for local\u002FDocker runs. External-deployment configs already define ",{"type":40,"tag":67,"props":6049,"children":6051},{"className":6050},[],[6052],{"type":46,"value":5986},{"type":46,"value":258},{"type":40,"tag":49,"props":6055,"children":6056},{},[6057,6062,6064,6069,6071,6077,6079,6085,6087,6093,6095,6101,6103,6108,6110,6115,6117,6123,6124,6130,6132,6137,6139,6144,6145,6150],{"type":40,"tag":125,"props":6058,"children":6059},{},[6060],{"type":46,"value":6061},"NEL env-var value prefixes (required):",{"type":46,"value":6063}," every value in an ",{"type":40,"tag":67,"props":6065,"children":6067},{"className":6066},[],[6068],{"type":46,"value":5049},{"type":46,"value":6070}," map needs an explicit prefix — ",{"type":40,"tag":67,"props":6072,"children":6074},{"className":6073},[],[6075],{"type":46,"value":6076},"host:VAR",{"type":46,"value":6078}," (read from the submitting shell's env at submit time), ",{"type":40,"tag":67,"props":6080,"children":6082},{"className":6081},[],[6083],{"type":46,"value":6084},"lit:value",{"type":46,"value":6086}," (literal string), or ",{"type":40,"tag":67,"props":6088,"children":6090},{"className":6089},[],[6091],{"type":46,"value":6092},"runtime:VAR",{"type":46,"value":6094}," (read in the job at run time). A bare value (e.g. ",{"type":40,"tag":67,"props":6096,"children":6098},{"className":6097},[],[6099],{"type":46,"value":6100},"DUMMY_API_KEY: dummy",{"type":46,"value":6102},") hard-errors: ",{"type":40,"tag":761,"props":6104,"children":6105},{},[6106],{"type":46,"value":6107},"\"Env var value '…' must have an explicit prefix.\"",{"type":46,"value":6109}," Use ",{"type":40,"tag":67,"props":6111,"children":6113},{"className":6112},[],[6114],{"type":46,"value":2797},{"type":46,"value":6116}," for constants like ",{"type":40,"tag":67,"props":6118,"children":6120},{"className":6119},[],[6121],{"type":46,"value":6122},"DUMMY_API_KEY",{"type":46,"value":782},{"type":40,"tag":67,"props":6125,"children":6127},{"className":6126},[],[6128],{"type":46,"value":6129},"VLLM_*",{"type":46,"value":6131}," backend selectors, ",{"type":40,"tag":67,"props":6133,"children":6135},{"className":6134},[],[6136],{"type":46,"value":1172},{"type":46,"value":6138}," for secrets like ",{"type":40,"tag":67,"props":6140,"children":6142},{"className":6141},[],[6143],{"type":46,"value":548},{"type":46,"value":744},{"type":40,"tag":67,"props":6146,"children":6148},{"className":6147},[],[6149],{"type":46,"value":555},{"type":46,"value":258},{"type":40,"tag":110,"props":6152,"children":6153},{},[],{"type":40,"tag":55,"props":6155,"children":6157},{"id":6156},"step-6-multi-node",[6158],{"type":46,"value":6159},"Step 6 — Multi-node",{"type":40,"tag":49,"props":6161,"children":6162},{},[6163,6165,6171],{"type":46,"value":6164},"For models > ~120B or higher throughput needs, read ",{"type":40,"tag":67,"props":6166,"children":6168},{"className":6167},[],[6169],{"type":46,"value":6170},"references\u002Fmulti-node.md",{"type":46,"value":6172}," for HAProxy multi-instance \u002F Ray TP\u002FPP \u002F combined patterns.",{"type":40,"tag":55,"props":6174,"children":6176},{"id":6175},"step-7-interceptors",[6177],{"type":46,"value":6178},"Step 7 — Interceptors",{"type":40,"tag":49,"props":6180,"children":6181},{},[6182,6184,6192,6194,6200,6201,6207,6209,6214,6216,6222],{"type":46,"value":6183},"Direct user to ",{"type":40,"tag":6185,"props":6186,"children":6190},"a",{"href":6187,"rel":6188},"https:\u002F\u002Fdocs.nvidia.com\u002Fnemo\u002Fevaluator\u002Flatest\u002Flibraries\u002Fnemo-evaluator\u002Finterceptors\u002Findex.html",[6189],"nofollow",[6191],{"type":46,"value":6187},{"type":46,"value":6193},". Do not provide generic interceptor info — read the specific interceptor's page if asked, then configure via ",{"type":40,"tag":67,"props":6195,"children":6197},{"className":6196},[],[6198],{"type":46,"value":6199},"evaluation.nemo_evaluator_config.target.api_endpoint.adapter_config",{"type":46,"value":736},{"type":40,"tag":67,"props":6202,"children":6204},{"className":6203},[],[6205],{"type":46,"value":6206},"target",{"type":46,"value":6208}," is a sibling of ",{"type":40,"tag":67,"props":6210,"children":6212},{"className":6211},[],[6213],{"type":46,"value":757},{"type":46,"value":6215},", not nested under it). Use the per-field syntax from the CLI Configuration section, not a full ",{"type":40,"tag":67,"props":6217,"children":6219},{"className":6218},[],[6220],{"type":46,"value":6221},"interceptors:",{"type":46,"value":6223}," list (that overrides the default chain).",{"type":40,"tag":49,"props":6225,"children":6226},{},[6227,6232,6234,6240,6241,6247,6249,6255,6256,6262],{"type":40,"tag":125,"props":6228,"children":6229},{},[6230],{"type":46,"value":6231},"Errata:",{"type":46,"value":6233}," Logging field names are ",{"type":40,"tag":67,"props":6235,"children":6237},{"className":6236},[],[6238],{"type":46,"value":6239},"max_logged_requests",{"type":46,"value":744},{"type":40,"tag":67,"props":6242,"children":6244},{"className":6243},[],[6245],{"type":46,"value":6246},"max_logged_responses",{"type":46,"value":6248}," (NOT ",{"type":40,"tag":67,"props":6250,"children":6252},{"className":6251},[],[6253],{"type":46,"value":6254},"max_saved_*",{"type":46,"value":744},{"type":40,"tag":67,"props":6257,"children":6259},{"className":6258},[],[6260],{"type":46,"value":6261},"max_*",{"type":46,"value":6263}," as some docs show).",{"type":40,"tag":55,"props":6265,"children":6267},{"id":6266},"step-75-container-registry-auth-slurm-private-images-only",[6268],{"type":46,"value":6269},"Step 7.5 — Container registry auth (SLURM private images only)",{"type":40,"tag":49,"props":6271,"children":6272},{},[6273],{"type":46,"value":6274},"Default images:",{"type":40,"tag":1519,"props":6276,"children":6277},{},[6278,6299],{"type":40,"tag":1523,"props":6279,"children":6280},{},[6281],{"type":40,"tag":1527,"props":6282,"children":6283},{},[6284,6289,6294],{"type":40,"tag":1531,"props":6285,"children":6286},{},[6287],{"type":46,"value":6288},"Framework",{"type":40,"tag":1531,"props":6290,"children":6291},{},[6292],{"type":46,"value":6293},"Image",{"type":40,"tag":1531,"props":6295,"children":6296},{},[6297],{"type":46,"value":6298},"Registry",{"type":40,"tag":1552,"props":6300,"children":6301},{},[6302,6332,6356,6377,6399],{"type":40,"tag":1527,"props":6303,"children":6304},{},[6305,6310,6327],{"type":40,"tag":1559,"props":6306,"children":6307},{},[6308],{"type":46,"value":6309},"vLLM",{"type":40,"tag":1559,"props":6311,"children":6312},{},[6313,6319,6321,6326],{"type":40,"tag":67,"props":6314,"children":6316},{"className":6315},[],[6317],{"type":46,"value":6318},"vllm\u002Fvllm-openai:v0.26.0",{"type":46,"value":6320}," (bump per recipe; never ",{"type":40,"tag":67,"props":6322,"children":6324},{"className":6323},[],[6325],{"type":46,"value":3561},{"type":46,"value":3172},{"type":40,"tag":1559,"props":6328,"children":6329},{},[6330],{"type":46,"value":6331},"DockerHub",{"type":40,"tag":1527,"props":6333,"children":6334},{},[6335,6340,6352],{"type":40,"tag":1559,"props":6336,"children":6337},{},[6338],{"type":46,"value":6339},"vLLM (NVFP4 on B300\u002FGB300)",{"type":40,"tag":1559,"props":6341,"children":6342},{},[6343,6345,6350],{"type":46,"value":6344},"default is already ",{"type":40,"tag":125,"props":6346,"children":6347},{},[6348],{"type":46,"value":6349},"CUDA-13",{"type":46,"value":6351},"; for older pins see Step 3",{"type":40,"tag":1559,"props":6353,"children":6354},{},[6355],{"type":46,"value":6331},{"type":40,"tag":1527,"props":6357,"children":6358},{},[6359,6364,6373],{"type":40,"tag":1559,"props":6360,"children":6361},{},[6362],{"type":46,"value":6363},"SGLang",{"type":40,"tag":1559,"props":6365,"children":6366},{},[6367],{"type":40,"tag":67,"props":6368,"children":6370},{"className":6369},[],[6371],{"type":46,"value":6372},"lmsysorg\u002Fsglang:latest",{"type":40,"tag":1559,"props":6374,"children":6375},{},[6376],{"type":46,"value":6331},{"type":40,"tag":1527,"props":6378,"children":6379},{},[6380,6385,6394],{"type":40,"tag":1559,"props":6381,"children":6382},{},[6383],{"type":46,"value":6384},"TRT-LLM",{"type":40,"tag":1559,"props":6386,"children":6387},{},[6388],{"type":40,"tag":67,"props":6389,"children":6391},{"className":6390},[],[6392],{"type":46,"value":6393},"nvcr.io\u002Fnvidia\u002Ftensorrt-llm\u002Frelease:...",{"type":40,"tag":1559,"props":6395,"children":6396},{},[6397],{"type":46,"value":6398},"NGC",{"type":40,"tag":1527,"props":6400,"children":6401},{},[6402,6407,6416],{"type":40,"tag":1559,"props":6403,"children":6404},{},[6405],{"type":46,"value":6406},"Eval tasks",{"type":40,"tag":1559,"props":6408,"children":6409},{},[6410],{"type":40,"tag":67,"props":6411,"children":6413},{"className":6412},[],[6414],{"type":46,"value":6415},"nvcr.io\u002Fnvidia\u002Feval-factory\u002F*:26.03",{"type":40,"tag":1559,"props":6417,"children":6418},{},[6419],{"type":46,"value":6398},{"type":40,"tag":951,"props":6421,"children":6422},{},[6423],{"type":40,"tag":49,"props":6424,"children":6425},{},[6426,6428,6432,6434,6439],{"type":46,"value":6427},"NVFP4 checkpoints on B300\u002FGB300 (sm_103) need a ",{"type":40,"tag":125,"props":6429,"children":6430},{},[6431],{"type":46,"value":6349},{"type":46,"value":6433}," image — CUDA-12 builds lack sm_103 FP4 kernels. The tag spelling depends on the vLLM version (Step 3 table); verify ",{"type":40,"tag":67,"props":6435,"children":6437},{"className":6436},[],[6438],{"type":46,"value":3692},{"type":46,"value":6440}," in your platform's child manifest.",{"type":40,"tag":49,"props":6442,"children":6443},{},[6444],{"type":46,"value":6445},"Public images → submit without preflight. Private\u002Frestricted → check credentials:",{"type":40,"tag":98,"props":6447,"children":6449},{"className":2046,"code":6448,"language":2048,"meta":103,"style":103},"ssh \u003Chost> \"grep -E '^\\s*machine\\s+' ~\u002F.config\u002Fenroot\u002F.credentials 2>\u002Fdev\u002Fnull\"\n",[6450],{"type":40,"tag":67,"props":6451,"children":6452},{"__ignoreMap":103},[6453],{"type":40,"tag":2054,"props":6454,"children":6455},{"class":2056,"line":2057},[6456,6460,6464,6468,6472,6476,6480,6485],{"type":40,"tag":2054,"props":6457,"children":6458},{"style":2061},[6459],{"type":46,"value":5411},{"type":40,"tag":2054,"props":6461,"children":6462},{"style":2082},[6463],{"type":46,"value":2085},{"type":40,"tag":2054,"props":6465,"children":6466},{"style":2066},[6467],{"type":46,"value":5420},{"type":40,"tag":2054,"props":6469,"children":6470},{"style":2093},[6471],{"type":46,"value":5425},{"type":40,"tag":2054,"props":6473,"children":6474},{"style":2082},[6475],{"type":46,"value":2100},{"type":40,"tag":2054,"props":6477,"children":6478},{"style":2082},[6479],{"type":46,"value":2417},{"type":40,"tag":2054,"props":6481,"children":6482},{"style":2066},[6483],{"type":46,"value":6484},"grep -E '^\\s*machine\\s+' ~\u002F.config\u002Fenroot\u002F.credentials 2>\u002Fdev\u002Fnull",{"type":40,"tag":2054,"props":6486,"children":6487},{"style":2082},[6488],{"type":46,"value":5443},{"type":40,"tag":49,"props":6490,"children":6491},{},[6492,6494,6500,6502,6508,6510,6515],{"type":46,"value":6493},"Add credentials per the common skill's ",{"type":40,"tag":67,"props":6495,"children":6497},{"className":6496},[],[6498],{"type":46,"value":6499},"slurm-setup.md",{"type":46,"value":6501}," §6 if missing. If you can't add, switch to a compatible public image (e.g. ",{"type":40,"tag":67,"props":6503,"children":6505},{"className":6504},[],[6506],{"type":46,"value":6507},"nvcr.io\u002Fnvidia\u002Fvllm:\u003CYY.MM>-py3",{"type":46,"value":6509}," — check catalog.ngc.nvidia.com). ",{"type":40,"tag":125,"props":6511,"children":6512},{},[6513],{"type":46,"value":6514},"Do not retry more than once",{"type":46,"value":6516}," after an auth failure.",{"type":40,"tag":110,"props":6518,"children":6519},{},[],{"type":40,"tag":55,"props":6521,"children":6523},{"id":6522},"step-8-run-evaluation-gated-dry-run-canary-full",[6524],{"type":46,"value":6525},"Step 8 — Run evaluation (gated dry-run → canary → full)",{"type":40,"tag":49,"props":6527,"children":6528},{},[6529],{"type":46,"value":6530},"Run directly when the user asked to launch; otherwise ask before submitting.",{"type":40,"tag":49,"props":6532,"children":6533},{},[6534,6539,6540,6545,6547,6552,6554,6558,6560,6565,6567,6571,6573,6578],{"type":40,"tag":125,"props":6535,"children":6536},{},[6537],{"type":46,"value":6538},"Env setup:",{"type":46,"value":774},{"type":40,"tag":67,"props":6541,"children":6543},{"className":6542},[],[6544],{"type":46,"value":317},{"type":46,"value":6546}," is normally already created and filled back in Step 1 (via ",{"type":40,"tag":67,"props":6548,"children":6550},{"className":6549},[],[6551],{"type":46,"value":293},{"type":46,"value":6553},"), at the ",{"type":40,"tag":125,"props":6555,"children":6556},{},[6557],{"type":46,"value":1137},{"type":46,"value":6559}," — the dir you run ",{"type":40,"tag":67,"props":6561,"children":6563},{"className":6562},[],[6564],{"type":46,"value":277},{"type":46,"value":6566}," from, not under the skill dir. Ensure it exists and source it — do ",{"type":40,"tag":125,"props":6568,"children":6569},{},[6570],{"type":46,"value":136},{"type":46,"value":6572}," clobber an existing ",{"type":40,"tag":67,"props":6574,"children":6576},{"className":6575},[],[6577],{"type":46,"value":317},{"type":46,"value":2829},{"type":40,"tag":98,"props":6580,"children":6582},{"className":2046,"code":6581,"language":2048,"meta":103,"style":103},"# .env lives at the workspace root (where you run nel); the template ships under the skill dir\n[ -f .env ] || cp \"$SKILL_DIR\u002Frecipes\u002Fenv.example\" .env   # create only if Step 1 didn't\nset -a && source .env && set +a\n\n# If pre_cmd\u002Fpost_cmd in config (review pre_cmd first — runs arbitrary commands):\nexport NEMO_EVALUATOR_TRUST_PRE_CMD=1\n# If nemo_skills.* + self-deployment, for LOCAL\u002FDocker runs only:\nexport DUMMY_API_KEY=dummy\n# On SLURM this shell export does NOT reach the container — instead declare\n# `DUMMY_API_KEY: lit:dummy` under evaluation.env_vars (see Step 5).\n",[6583],{"type":40,"tag":67,"props":6584,"children":6585},{"__ignoreMap":103},[6586,6594,6654,6694,6703,6711,6734,6742,6763,6771],{"type":40,"tag":2054,"props":6587,"children":6588},{"class":2056,"line":2057},[6589],{"type":40,"tag":2054,"props":6590,"children":6591},{"style":4070},[6592],{"type":46,"value":6593},"# .env lives at the workspace root (where you run nel); the template ships under the skill dir\n",{"type":40,"tag":2054,"props":6595,"children":6596},{"class":2056,"line":2465},[6597,6602,6607,6612,6616,6621,6626,6630,6635,6640,6644,6649],{"type":40,"tag":2054,"props":6598,"children":6599},{"style":2082},[6600],{"type":46,"value":6601},"[",{"type":40,"tag":2054,"props":6603,"children":6604},{"style":2082},[6605],{"type":46,"value":6606}," -f",{"type":40,"tag":2054,"props":6608,"children":6609},{"style":2093},[6610],{"type":46,"value":6611}," .env ",{"type":40,"tag":2054,"props":6613,"children":6614},{"style":2082},[6615],{"type":46,"value":2190},{"type":40,"tag":2054,"props":6617,"children":6618},{"style":2082},[6619],{"type":46,"value":6620}," ||",{"type":40,"tag":2054,"props":6622,"children":6623},{"style":2061},[6624],{"type":46,"value":6625}," cp",{"type":40,"tag":2054,"props":6627,"children":6628},{"style":2082},[6629],{"type":46,"value":2417},{"type":40,"tag":2054,"props":6631,"children":6632},{"style":2093},[6633],{"type":46,"value":6634},"$SKILL_DIR",{"type":40,"tag":2054,"props":6636,"children":6637},{"style":2066},[6638],{"type":46,"value":6639},"\u002Frecipes\u002Fenv.example",{"type":40,"tag":2054,"props":6641,"children":6642},{"style":2082},[6643],{"type":46,"value":2427},{"type":40,"tag":2054,"props":6645,"children":6646},{"style":2066},[6647],{"type":46,"value":6648}," .env",{"type":40,"tag":2054,"props":6650,"children":6651},{"style":4070},[6652],{"type":46,"value":6653},"   # create only if Step 1 didn't\n",{"type":40,"tag":2054,"props":6655,"children":6656},{"class":2056,"line":2474},[6657,6662,6667,6671,6676,6680,6684,6689],{"type":40,"tag":2054,"props":6658,"children":6659},{"style":4866},[6660],{"type":46,"value":6661},"set",{"type":40,"tag":2054,"props":6663,"children":6664},{"style":2066},[6665],{"type":46,"value":6666}," -a",{"type":40,"tag":2054,"props":6668,"children":6669},{"style":2082},[6670],{"type":46,"value":4759},{"type":40,"tag":2054,"props":6672,"children":6673},{"style":4866},[6674],{"type":46,"value":6675}," source",{"type":40,"tag":2054,"props":6677,"children":6678},{"style":2066},[6679],{"type":46,"value":6648},{"type":40,"tag":2054,"props":6681,"children":6682},{"style":2082},[6683],{"type":46,"value":4759},{"type":40,"tag":2054,"props":6685,"children":6686},{"style":4866},[6687],{"type":46,"value":6688}," set",{"type":40,"tag":2054,"props":6690,"children":6691},{"style":2066},[6692],{"type":46,"value":6693}," +a\n",{"type":40,"tag":2054,"props":6695,"children":6696},{"class":2056,"line":2483},[6697],{"type":40,"tag":2054,"props":6698,"children":6700},{"emptyLinePlaceholder":6699},true,[6701],{"type":46,"value":6702},"\n",{"type":40,"tag":2054,"props":6704,"children":6705},{"class":2056,"line":2492},[6706],{"type":40,"tag":2054,"props":6707,"children":6708},{"style":4070},[6709],{"type":46,"value":6710},"# If pre_cmd\u002Fpost_cmd in config (review pre_cmd first — runs arbitrary commands):\n",{"type":40,"tag":2054,"props":6712,"children":6713},{"class":2056,"line":3378},[6714,6719,6724,6729],{"type":40,"tag":2054,"props":6715,"children":6716},{"style":3348},[6717],{"type":46,"value":6718},"export",{"type":40,"tag":2054,"props":6720,"children":6721},{"style":2093},[6722],{"type":46,"value":6723}," NEMO_EVALUATOR_TRUST_PRE_CMD",{"type":40,"tag":2054,"props":6725,"children":6726},{"style":2082},[6727],{"type":46,"value":6728},"=",{"type":40,"tag":2054,"props":6730,"children":6731},{"style":4088},[6732],{"type":46,"value":6733},"1\n",{"type":40,"tag":2054,"props":6735,"children":6736},{"class":2056,"line":3387},[6737],{"type":40,"tag":2054,"props":6738,"children":6739},{"style":4070},[6740],{"type":46,"value":6741},"# If nemo_skills.* + self-deployment, for LOCAL\u002FDocker runs only:\n",{"type":40,"tag":2054,"props":6743,"children":6744},{"class":2056,"line":3396},[6745,6749,6754,6758],{"type":40,"tag":2054,"props":6746,"children":6747},{"style":3348},[6748],{"type":46,"value":6718},{"type":40,"tag":2054,"props":6750,"children":6751},{"style":2093},[6752],{"type":46,"value":6753}," DUMMY_API_KEY",{"type":40,"tag":2054,"props":6755,"children":6756},{"style":2082},[6757],{"type":46,"value":6728},{"type":40,"tag":2054,"props":6759,"children":6760},{"style":2093},[6761],{"type":46,"value":6762},"dummy\n",{"type":40,"tag":2054,"props":6764,"children":6765},{"class":2056,"line":3405},[6766],{"type":40,"tag":2054,"props":6767,"children":6768},{"style":4070},[6769],{"type":46,"value":6770},"# On SLURM this shell export does NOT reach the container — instead declare\n",{"type":40,"tag":2054,"props":6772,"children":6774},{"class":2056,"line":6773},10,[6775],{"type":40,"tag":2054,"props":6776,"children":6777},{"style":4070},[6778],{"type":46,"value":6779},"# `DUMMY_API_KEY: lit:dummy` under evaluation.env_vars (see Step 5).\n",{"type":40,"tag":49,"props":6781,"children":6782},{},[6783,6788],{"type":40,"tag":125,"props":6784,"children":6785},{},[6786],{"type":46,"value":6787},"Step 8.1 — Dry-run",{"type":46,"value":6789}," (config validation):",{"type":40,"tag":98,"props":6791,"children":6793},{"className":2046,"code":6792,"language":2048,"meta":103,"style":103},"nel run --config \u003Cpath> --dry-run\n",[6794],{"type":40,"tag":67,"props":6795,"children":6796},{"__ignoreMap":103},[6797],{"type":40,"tag":2054,"props":6798,"children":6799},{"class":2056,"line":2057},[6800,6804,6809,6814,6818,6823,6828,6832],{"type":40,"tag":2054,"props":6801,"children":6802},{"style":2061},[6803],{"type":46,"value":277},{"type":40,"tag":2054,"props":6805,"children":6806},{"style":2066},[6807],{"type":46,"value":6808}," run",{"type":40,"tag":2054,"props":6810,"children":6811},{"style":2066},[6812],{"type":46,"value":6813}," --config",{"type":40,"tag":2054,"props":6815,"children":6816},{"style":2082},[6817],{"type":46,"value":2085},{"type":40,"tag":2054,"props":6819,"children":6820},{"style":2066},[6821],{"type":46,"value":6822},"pat",{"type":40,"tag":2054,"props":6824,"children":6825},{"style":2093},[6826],{"type":46,"value":6827},"h",{"type":40,"tag":2054,"props":6829,"children":6830},{"style":2082},[6831],{"type":46,"value":2100},{"type":40,"tag":2054,"props":6833,"children":6834},{"style":2066},[6835],{"type":46,"value":6836}," --dry-run\n",{"type":40,"tag":49,"props":6838,"children":6839},{},[6840,6842,6847],{"type":46,"value":6841},"Fix unresolved ",{"type":40,"tag":67,"props":6843,"children":6845},{"className":6844},[],[6846],{"type":46,"value":874},{"type":46,"value":6848},", bad Hydra overrides, missing env vars, invalid mounts, image issues, sbatch errors, obvious deployment errors before proceeding.",{"type":40,"tag":951,"props":6850,"children":6851},{},[6852],{"type":40,"tag":49,"props":6853,"children":6854},{},[6855,6860,6862,6868,6870,6875],{"type":40,"tag":125,"props":6856,"children":6857},{},[6858],{"type":46,"value":6859},"Dry-run does NOT validate the image\u002FvLLM version",{"type":46,"value":6861}," (image pulled only at deploy). Confirm ",{"type":40,"tag":67,"props":6863,"children":6865},{"className":6864},[],[6866],{"type":46,"value":6867},"image:",{"type":46,"value":6869}," ≥ the exact model's ",{"type":40,"tag":67,"props":6871,"children":6873},{"className":6872},[],[6874],{"type":46,"value":3514},{"type":46,"value":6876}," minimum (Step 3) before submitting — too-old passes dry-run, then crashes mid-inference.",{"type":40,"tag":951,"props":6878,"children":6879},{},[6880],{"type":40,"tag":49,"props":6881,"children":6882},{},[6883,6888,6890,6896,6897,6903,6905,6910,6912,6917,6919,6924],{"type":40,"tag":125,"props":6884,"children":6885},{},[6886],{"type":46,"value":6887},"Non-fatal noise:",{"type":46,"value":6889}," \"Failed to get manifest\"\u002F",{"type":40,"tag":67,"props":6891,"children":6893},{"className":6892},[],[6894],{"type":46,"value":6895},"401",{"type":46,"value":185},{"type":40,"tag":67,"props":6898,"children":6900},{"className":6899},[],[6901],{"type":46,"value":6902},"404",{"type":46,"value":6904},", \"Could not extract frame definition file\", \"proceeding with minimal task definition\", \"Found N unlisted task(s)\" — expected for ",{"type":40,"tag":67,"props":6906,"children":6908},{"className":6907},[],[6909],{"type":46,"value":5864},{"type":46,"value":6911},"\u002Frecipe tasks and private (gitlab) containers; the task still runs in-container. Set ",{"type":40,"tag":67,"props":6913,"children":6915},{"className":6914},[],[6916],{"type":46,"value":842},{"type":46,"value":6918},". Real blockers: unresolved ",{"type":40,"tag":67,"props":6920,"children":6922},{"className":6921},[],[6923],{"type":46,"value":874},{"type":46,"value":6925},", interpolation errors, bad mounts, sbatch rejections.",{"type":40,"tag":49,"props":6927,"children":6928},{},[6929,6934],{"type":40,"tag":125,"props":6930,"children":6931},{},[6932],{"type":46,"value":6933},"Step 8.2 — Canary",{"type":46,"value":6935}," (limited-samples, validates everything dry-run can't):",{"type":40,"tag":98,"props":6937,"children":6939},{"className":2046,"code":6938,"language":2048,"meta":103,"style":103},"nel run --config \u003Cpath> -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10\n",[6940],{"type":40,"tag":67,"props":6941,"children":6942},{"__ignoreMap":103},[6943],{"type":40,"tag":2054,"props":6944,"children":6945},{"class":2056,"line":2057},[6946,6950,6954,6958,6962,6966,6970,6974,6979,6984],{"type":40,"tag":2054,"props":6947,"children":6948},{"style":2061},[6949],{"type":46,"value":277},{"type":40,"tag":2054,"props":6951,"children":6952},{"style":2066},[6953],{"type":46,"value":6808},{"type":40,"tag":2054,"props":6955,"children":6956},{"style":2066},[6957],{"type":46,"value":6813},{"type":40,"tag":2054,"props":6959,"children":6960},{"style":2082},[6961],{"type":46,"value":2085},{"type":40,"tag":2054,"props":6963,"children":6964},{"style":2066},[6965],{"type":46,"value":6822},{"type":40,"tag":2054,"props":6967,"children":6968},{"style":2093},[6969],{"type":46,"value":6827},{"type":40,"tag":2054,"props":6971,"children":6972},{"style":2082},[6973],{"type":46,"value":2100},{"type":40,"tag":2054,"props":6975,"children":6976},{"style":2066},[6977],{"type":46,"value":6978}," -o",{"type":40,"tag":2054,"props":6980,"children":6981},{"style":2066},[6982],{"type":46,"value":6983}," ++evaluation.nemo_evaluator_config.config.params.limit_samples=",{"type":40,"tag":2054,"props":6985,"children":6986},{"style":4088},[6987],{"type":46,"value":6988},"10\n",{"type":40,"tag":49,"props":6990,"children":6991},{},[6992],{"type":46,"value":6993},"Catches judge auth\u002Frate-limits, container failures, sandbox issues, OOM, bad request formatting, low evaluated counts. Always inspect logs:",{"type":40,"tag":98,"props":6995,"children":6997},{"className":2046,"code":6996,"language":2048,"meta":103,"style":103},"nel status \u003Cid>\nnel info \u003Cid> --logs\nssh \u003Cuser>@\u003Chost> \"grep -i 'traceback\\|exception\\|error\\|failed\\|oom\\|killed\\|timeout\\|unauthorized\\|rate limit\\|sandbox\\|container\\|judge\\|parse\\|scoring' \u003Clog_path>\u002F*.log\"\n",[6998],{"type":40,"tag":67,"props":6999,"children":7000},{"__ignoreMap":103},[7001,7032,7065],{"type":40,"tag":2054,"props":7002,"children":7003},{"class":2056,"line":2057},[7004,7008,7013,7017,7022,7027],{"type":40,"tag":2054,"props":7005,"children":7006},{"style":2061},[7007],{"type":46,"value":277},{"type":40,"tag":2054,"props":7009,"children":7010},{"style":2066},[7011],{"type":46,"value":7012}," status",{"type":40,"tag":2054,"props":7014,"children":7015},{"style":2082},[7016],{"type":46,"value":2085},{"type":40,"tag":2054,"props":7018,"children":7019},{"style":2066},[7020],{"type":46,"value":7021},"i",{"type":40,"tag":2054,"props":7023,"children":7024},{"style":2093},[7025],{"type":46,"value":7026},"d",{"type":40,"tag":2054,"props":7028,"children":7029},{"style":2082},[7030],{"type":46,"value":7031},">\n",{"type":40,"tag":2054,"props":7033,"children":7034},{"class":2056,"line":2465},[7035,7039,7044,7048,7052,7056,7060],{"type":40,"tag":2054,"props":7036,"children":7037},{"style":2061},[7038],{"type":46,"value":277},{"type":40,"tag":2054,"props":7040,"children":7041},{"style":2066},[7042],{"type":46,"value":7043}," info",{"type":40,"tag":2054,"props":7045,"children":7046},{"style":2082},[7047],{"type":46,"value":2085},{"type":40,"tag":2054,"props":7049,"children":7050},{"style":2066},[7051],{"type":46,"value":7021},{"type":40,"tag":2054,"props":7053,"children":7054},{"style":2093},[7055],{"type":46,"value":7026},{"type":40,"tag":2054,"props":7057,"children":7058},{"style":2082},[7059],{"type":46,"value":2100},{"type":40,"tag":2054,"props":7061,"children":7062},{"style":2066},[7063],{"type":46,"value":7064}," --logs\n",{"type":40,"tag":2054,"props":7066,"children":7067},{"class":2056,"line":2474},[7068,7072,7076,7081,7086,7090,7095,7099,7103,7107,7111,7115,7120],{"type":40,"tag":2054,"props":7069,"children":7070},{"style":2061},[7071],{"type":46,"value":5411},{"type":40,"tag":2054,"props":7073,"children":7074},{"style":2082},[7075],{"type":46,"value":2085},{"type":40,"tag":2054,"props":7077,"children":7078},{"style":2066},[7079],{"type":46,"value":7080},"use",{"type":40,"tag":2054,"props":7082,"children":7083},{"style":2093},[7084],{"type":46,"value":7085},"r",{"type":40,"tag":2054,"props":7087,"children":7088},{"style":2082},[7089],{"type":46,"value":2100},{"type":40,"tag":2054,"props":7091,"children":7092},{"style":2066},[7093],{"type":46,"value":7094},"@",{"type":40,"tag":2054,"props":7096,"children":7097},{"style":2082},[7098],{"type":46,"value":2173},{"type":40,"tag":2054,"props":7100,"children":7101},{"style":2066},[7102],{"type":46,"value":5420},{"type":40,"tag":2054,"props":7104,"children":7105},{"style":2093},[7106],{"type":46,"value":5425},{"type":40,"tag":2054,"props":7108,"children":7109},{"style":2082},[7110],{"type":46,"value":2100},{"type":40,"tag":2054,"props":7112,"children":7113},{"style":2082},[7114],{"type":46,"value":2417},{"type":40,"tag":2054,"props":7116,"children":7117},{"style":2066},[7118],{"type":46,"value":7119},"grep -i 'traceback\\|exception\\|error\\|failed\\|oom\\|killed\\|timeout\\|unauthorized\\|rate limit\\|sandbox\\|container\\|judge\\|parse\\|scoring' \u003Clog_path>\u002F*.log",{"type":40,"tag":2054,"props":7121,"children":7122},{"style":2082},[7123],{"type":46,"value":5443},{"type":40,"tag":49,"props":7125,"children":7126},{},[7127,7129,7134,7136,7141,7142,7147,7149,7154],{"type":46,"value":7128},"Canary each risky task class separately (judge-scored, code-execution, model-only). Start ",{"type":40,"tag":67,"props":7130,"children":7132},{"className":7131},[],[7133],{"type":46,"value":3903},{"type":46,"value":7135}," conservatively; raise only after judge\u002Fsandbox logs are clean — they bottleneck before the model. For capacity-bound runs, tune ",{"type":40,"tag":67,"props":7137,"children":7139},{"className":7138},[],[7140],{"type":46,"value":3903},{"type":46,"value":185},{"type":40,"tag":67,"props":7143,"children":7145},{"className":7144},[],[7146],{"type":46,"value":5132},{"type":46,"value":7148}," here against vLLM's reported max concurrency + preemption — see ",{"type":40,"tag":67,"props":7150,"children":7152},{"className":7151},[],[7153],{"type":46,"value":3506},{"type":46,"value":258},{"type":40,"tag":49,"props":7156,"children":7157},{},[7158,7160,7166,7168,7174],{"type":46,"value":7159},"Single-task rerun: ",{"type":40,"tag":67,"props":7161,"children":7163},{"className":7162},[],[7164],{"type":46,"value":7165},"nel run --config \u003Cpath> -t \u003Ctask_name>",{"type":46,"value":7167}," (combine with ",{"type":40,"tag":67,"props":7169,"children":7171},{"className":7170},[],[7172],{"type":46,"value":7173},"-o ++...limit_samples=10",{"type":46,"value":7175}," for canary).",{"type":40,"tag":49,"props":7177,"children":7178},{},[7179,7184],{"type":40,"tag":125,"props":7180,"children":7181},{},[7182],{"type":46,"value":7183},"Step 8.3 — Full run",{"type":46,"value":7185}," (after canary passes):",{"type":40,"tag":98,"props":7187,"children":7189},{"className":2046,"code":7188,"language":2048,"meta":103,"style":103},"nel run --config \u003Cpath>\n",[7190],{"type":40,"tag":67,"props":7191,"children":7192},{"__ignoreMap":103},[7193],{"type":40,"tag":2054,"props":7194,"children":7195},{"class":2056,"line":2057},[7196,7200,7204,7208,7212,7216,7220],{"type":40,"tag":2054,"props":7197,"children":7198},{"style":2061},[7199],{"type":46,"value":277},{"type":40,"tag":2054,"props":7201,"children":7202},{"style":2066},[7203],{"type":46,"value":6808},{"type":40,"tag":2054,"props":7205,"children":7206},{"style":2066},[7207],{"type":46,"value":6813},{"type":40,"tag":2054,"props":7209,"children":7210},{"style":2082},[7211],{"type":46,"value":2085},{"type":40,"tag":2054,"props":7213,"children":7214},{"style":2066},[7215],{"type":46,"value":6822},{"type":40,"tag":2054,"props":7217,"children":7218},{"style":2093},[7219],{"type":46,"value":6827},{"type":40,"tag":2054,"props":7221,"children":7222},{"style":2082},[7223],{"type":46,"value":7031},{"type":40,"tag":49,"props":7225,"children":7226},{},[7227,7229,7234],{"type":46,"value":7228},"Remove ",{"type":40,"tag":67,"props":7230,"children":7232},{"className":7231},[],[7233],{"type":46,"value":631},{"type":46,"value":7235}," overrides; keep canary-validated parallelism. If the canary fails, fix and rerun the canary — don't skip to full.",{"type":40,"tag":49,"props":7237,"children":7238},{},[7239,7244,7246,7251,7253,7258,7260,7265,7267,7272],{"type":40,"tag":125,"props":7240,"children":7241},{},[7242],{"type":46,"value":7243},"Monitoring:",{"type":46,"value":7245}," Register the job per the ",{"type":40,"tag":125,"props":7247,"children":7248},{},[7249],{"type":46,"value":7250},"monitor",{"type":46,"value":7252}," skill for cross-session tracking. One-off live status \u002F debugging → ",{"type":40,"tag":125,"props":7254,"children":7255},{},[7256],{"type":46,"value":7257},"launching-evals",{"type":46,"value":7259}," skill. Past-run MLflow queries → ",{"type":40,"tag":125,"props":7261,"children":7262},{},[7263],{"type":46,"value":7264},"accessing-mlflow",{"type":46,"value":7266}," skill. NEL timeout\u002Fresume → read ",{"type":40,"tag":67,"props":7268,"children":7270},{"className":7269},[],[7271],{"type":46,"value":4402},{"type":46,"value":7273}," before treating the run as failed.",{"type":40,"tag":110,"props":7275,"children":7276},{},[],{"type":40,"tag":55,"props":7278,"children":7280},{"id":7279},"step-9-verify-completed-run",[7281],{"type":46,"value":7282},"Step 9 — Verify completed run",{"type":40,"tag":49,"props":7284,"children":7285},{},[7286,7288,7293,7295,7300,7302,7308],{"type":46,"value":7287},"Before pulling\u002Freporting scores, validate the run. Read ",{"type":40,"tag":67,"props":7289,"children":7291},{"className":7290},[],[7292],{"type":46,"value":4402},{"type":46,"value":7294}," for NEL timeout\u002Fresume behavior, completed-run validation, diagnostics, and score harvesting. For a baseline that will be compared with a candidate, also perform its ",{"type":40,"tag":125,"props":7296,"children":7297},{},[7298],{"type":46,"value":7299},"External Baseline Sanity Check",{"type":46,"value":7301}," before a success verdict, then hand the validated runs to ",{"type":40,"tag":67,"props":7303,"children":7305},{"className":7304},[],[7306],{"type":46,"value":7307},"compare-results",{"type":46,"value":7309}," for baseline-vs-candidate deltas.",{"type":40,"tag":110,"props":7311,"children":7312},{},[],{"type":40,"tag":49,"props":7314,"children":7315},{},[7316,7318,7324,7326],{"type":46,"value":7317},"Issues: ",{"type":40,"tag":6185,"props":7319,"children":7322},{"href":7320,"rel":7321},"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Fissues",[6189],[7323],{"type":46,"value":7320},{"type":46,"value":7325}," · ",{"type":40,"tag":6185,"props":7327,"children":7330},{"href":7328,"rel":7329},"https:\u002F\u002Fgithub.com\u002FNVIDIA-NeMo\u002FEvaluator\u002Fdiscussions",[6189],[7331],{"type":46,"value":7328},{"type":40,"tag":7333,"props":7334,"children":7335},"style",{},[7336],{"type":46,"value":7337},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":7339,"total":7497},[7340,7358,7375,7386,7398,7412,7425,7439,7452,7463,7477,7486],{"slug":7341,"name":7341,"fn":7342,"description":7343,"org":7344,"tags":7345,"stars":7355,"repoUrl":7356,"updatedAt":7357},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7346,7349,7352],{"name":7347,"slug":7348,"type":15},"Documentation","documentation",{"name":7350,"slug":7351,"type":15},"MCP","mcp",{"name":7353,"slug":7354,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":7359,"name":7359,"fn":7360,"description":7361,"org":7362,"tags":7363,"stars":7372,"repoUrl":7373,"updatedAt":7374},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7364,7367,7369],{"name":7365,"slug":7366,"type":15},"Containers","containers",{"name":7368,"slug":3322,"type":15},"Deployment",{"name":7370,"slug":7371,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":7376,"name":7376,"fn":7377,"description":7378,"org":7379,"tags":7380,"stars":7372,"repoUrl":7373,"updatedAt":7385},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7381,7384],{"name":7382,"slug":7383,"type":15},"CI\u002FCD","ci-cd",{"name":7368,"slug":3322,"type":15},"2026-07-14T05:25:59.97109",{"slug":7387,"name":7387,"fn":7388,"description":7389,"org":7390,"tags":7391,"stars":7372,"repoUrl":7373,"updatedAt":7397},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7392,7393,7394],{"name":7382,"slug":7383,"type":15},{"name":7368,"slug":3322,"type":15},{"name":7395,"slug":7396,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":7399,"name":7399,"fn":7400,"description":7401,"org":7402,"tags":7403,"stars":7372,"repoUrl":7373,"updatedAt":7411},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7404,7407,7408],{"name":7405,"slug":7406,"type":15},"Debugging","debugging",{"name":7395,"slug":7396,"type":15},{"name":7409,"slug":7410,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":7413,"name":7413,"fn":7414,"description":7415,"org":7416,"tags":7417,"stars":7372,"repoUrl":7373,"updatedAt":7424},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7418,7421],{"name":7419,"slug":7420,"type":15},"Best Practices","best-practices",{"name":7422,"slug":7423,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":7426,"name":7426,"fn":7427,"description":7428,"org":7429,"tags":7430,"stars":7372,"repoUrl":7373,"updatedAt":7438},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7431,7434,7437],{"name":7432,"slug":7433,"type":15},"Machine Learning","machine-learning",{"name":7435,"slug":7436,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":7440,"name":7440,"fn":7441,"description":7442,"org":7443,"tags":7444,"stars":7372,"repoUrl":7373,"updatedAt":7451},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7445,7448],{"name":7446,"slug":7447,"type":15},"QA","qa",{"name":7449,"slug":7450,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":7453,"name":7453,"fn":7454,"description":7455,"org":7456,"tags":7457,"stars":7372,"repoUrl":7373,"updatedAt":7462},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7458,7459],{"name":7368,"slug":3322,"type":15},{"name":7460,"slug":7461,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":7464,"name":7464,"fn":7465,"description":7466,"org":7467,"tags":7468,"stars":7372,"repoUrl":7373,"updatedAt":7476},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7469,7472,7473],{"name":7470,"slug":7471,"type":15},"Code Review","code-review",{"name":7395,"slug":7396,"type":15},{"name":7474,"slug":7475,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":7478,"name":7478,"fn":7479,"description":7480,"org":7481,"tags":7482,"stars":7372,"repoUrl":7373,"updatedAt":7485},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7483,7484],{"name":7446,"slug":7447,"type":15},{"name":7449,"slug":7450,"type":15},"2026-07-14T05:25:54.928983",{"slug":7487,"name":7487,"fn":7488,"description":7489,"org":7490,"tags":7491,"stars":7372,"repoUrl":7373,"updatedAt":7496},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7492,7495],{"name":7493,"slug":7494,"type":15},"Automation","automation",{"name":7382,"slug":7383,"type":15},"2026-07-30T05:29:03.275638",563,{"items":7499,"total":2474},[7500,7511,7522],{"slug":7501,"name":7501,"fn":7502,"description":7503,"org":7504,"tags":7505,"stars":23,"repoUrl":24,"updatedAt":7510},"day0-release","automate quantized checkpoint releases","Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT \u002F REGRESSION \u002F ANOMALOUS \u002F INFEASIBLE). Use when the user asks to \"release a model at day-0\", \"quantize and validate model X is within N% of baseline and tell me if it's publishable\", or \"run the full day-0 workflow\". Do NOT use for single-stage requests — quantizing only (use ptq), serving only (use deployment), evaluating only (use evaluation), or comparing two existing runs (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7506,7507,7508,7509],{"name":7493,"slug":7494,"type":15},{"name":7382,"slug":7383,"type":15},{"name":7368,"slug":3322,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:18:55.832925",{"slug":3322,"name":3322,"fn":7512,"description":7513,"org":7514,"tags":7515,"stars":23,"repoUrl":24,"updatedAt":7521},"deploy LLM checkpoints as API endpoints","Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7516,7517,7518,7519],{"name":7368,"slug":3322,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":6309,"slug":7520,"type":15},"vllm","2026-08-31T09:18:55.144088",{"slug":4,"name":4,"fn":5,"description":6,"org":7523,"tags":7524,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[7525,7526,7527,7528],{"name":13,"slug":14,"type":15},{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15}]