[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nvshmem-collect-performance-data":3,"mdc--tw3z2v-key":36,"related-repo-nvidia-nvshmem-collect-performance-data":1424,"related-org-nvidia-nvshmem-collect-performance-data":1507},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":31,"sourceUrl":34,"mdContent":35},"nvshmem-collect-performance-data","collect NVSHMEM performance metrics","Collect and package NVSHMEM put\u002Fget bandwidth, latency, and other perftest results with system and topology evidence for performance sanity checks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Monitoring","monitoring",{"name":9,"slug":8,"type":15},579,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvshmem","2026-08-31T09:19:00.917887","Apache-2.0",104,[26,27,28,29,8,30],"communciations","cpp","cuda","deep-learning","python",{"repoUrl":21,"stars":20,"forks":24,"topics":32,"description":33},[26,27,28,29,8,30],"NVIDIA NVSHMEM is a parallel programming interface for NVIDIA GPUs based on OpenSHMEM. NVSHMEM can significantly reduce multi-process communication and coordination overheads by allowing programmers to perform one-sided communication from within CUDA kernels and on CUDA streams.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvshmem\u002Ftree\u002FHEAD\u002Fskills\u002Fnvshmem-collect-performance-data","---\nname: nvshmem-collect-performance-data\ndescription: Collect and package NVSHMEM put\u002Fget bandwidth, latency, and other perftest results with system and topology evidence for performance sanity checks.\nlicense: Apache-2.0\nmetadata:\n  version: \"1.0.0\"\n  author: NVIDIA NVSHMEM Team \u003Cnvshmem@nvidia.com>\n  tags:\n    - nvshmem\n    - performance\n    - perftest\n---\n\n# NVSHMEM Performance Data Collection\n\n## Purpose\n\nCollect reproducible NVSHMEM performance evidence, with the four device put\u002Fget bandwidth and latency tests as the recommended sanity suite. Preserve the system, topology, placement, initialization, transport, commands, raw output, and tuning configuration in a portable Markdown report without inventing system-specific acceptance thresholds.\n\n## Requirements\n\n- Use an allocated compute node with the target GPUs visible; never collect evidence or run benchmarks on a login node.\n- Make the target NVSHMEM installation, selected perftest executable, and a known-good launcher\u002Fbootstrap path available. Direct target access is optional when a user can run the staged commands and return their complete output.\n- Provide a writable output parent directory for the artifact package. The bundled collector itself is read-only and writes only to stdout.\n- Require no API keys, privileged access, package installation, or system-configuration changes.\n\n## Inputs\n\n### Required\n\nResolve these values from the request, the target environment, or focused follow-up before benchmark execution:\n\n- an explicit test, preset, or suite selection;\n- the NVSHMEM installation prefix and perftest root;\n- access to the participating compute nodes, or a user who can run and return the staged commands;\n- a known-good launcher and bootstrap path;\n- allocated nodes, total PEs, PEs per node, and local PE-to-GPU placement;\n- the output parent directory for a new artifact package.\n\nFor an unspecified request, require only the suite choice first. Do not collect the remaining execution inputs until the user selects a suite. Do not ask again for values that are already explicit or safely observable.\n\n### Optional\n\n- exact message-size ranges, datatype, atomic operation, thread-group scope, CUDA graph, mmap, EGM, bidirectional, or other selected-test options;\n- an installation or source tree containing the selected test's standard `.args` file;\n- a caller-provided launcher command, binding policy, artifact-directory name, or reporting notes;\n- an identical matched baseline for range assessment;\n- raw stdout, stderr, exit statuses, hostname output, and environment evidence from user-run collection when direct execution is unavailable.\n\n## Scripts\n\n| Script | Use | Invocation |\n| --- | --- | --- |\n| `scripts\u002Fcollect-performance-environment.sh` | Collect read-only NVSHMEM, GPU, network, RDMA, launcher, topology, and allowlisted environment evidence. It writes to stdout and can exit with status 2 after producing useful partial output. | Resolve the script from this skill directory and use `run_script` on every participating compute node. If `run_script` is unavailable, run `bash scripts\u002Fcollect-performance-environment.sh`; retain complete output even when it exits nonzero. |\n\n## Instructions\n\n### Resolve the Request\n\nClassify the request before running a benchmark.\n\n- Treat an exact executable, named family, or named preset suite as an explicit selection.\n- Treat requests such as \"run perftests,\" \"check NVSHMEM performance,\" or \"perform a sanity check\" as unspecified.\n- For an unspecified request, read [references\u002Fperftest-catalog.md](references\u002Fperftest-catalog.md), summarize the available families, recommend the default device sanity suite, and require the user to choose one of:\n  1. the recommended four-test device sanity suite;\n  2. a custom device suite;\n  3. a host or feature-specific suite.\n- Run nothing until the user explicitly selects a choice. After a custom category is chosen, ask only for the exact tests, placement, PE count, and special mode that materially affect the run.\n- If the user already selected a test or suite, skip the menu and honor that scope.\n- Use canonical executable names. Do not translate or accept shortened `_lat` names.\n\nAssume the standard NVSHMEM perftest set exists. Do not inventory executables or enumerate every argument variant before selection. When a selected executable is missing at launch, report an installation or build problem instead of silently substituting another test.\n\n### Default Device Sanity Suite\n\nUse exactly these device point-to-point tests when the user selects the recommended suite:\n\n```text\ndevice\u002Fpt-to-pt\u002Fshmem_put_bw\ndevice\u002Fpt-to-pt\u002Fshmem_put_latency\ndevice\u002Fpt-to-pt\u002Fshmem_get_bw\ndevice\u002Fpt-to-pt\u002Fshmem_get_latency\n```\n\nRun each test with exactly two PEs. Measure one same-node placement and, when a two-node allocation is already available, one two-node placement with one PE per node. Do not request or create a new allocation merely to add the two-node placement; record it as unavailable.\n\nFor a selected non-default test, read its entry in [references\u002Fperftest-catalog.md](references\u002Fperftest-catalog.md). Use two PEs for point-to-point tests unless the benchmark documents another requirement. Require an explicit node count, PE count, PEs per node, and GPU placement for collective, tile, or allocation tests.\n\n### Prepare the Target\n\nResolve these values from the user, environment, or direct inspection:\n\n- NVSHMEM installation prefix and perftest root;\n- launcher and bootstrap path;\n- allocated nodes and task\u002FGPU capacity;\n- local PE-to-GPU binding;\n- selected same-node and two-node placements;\n- output parent directory.\n\nResolve `PERF_ROOT` from an explicit path first, then `$NVSHMEM_PREFIX\u002Fbin\u002Fperftest`, then `$NVSHMEM_HOME\u002Fbin\u002Fperftest`. Do not search the filesystem for alternate installations. Check only the selected executable paths with `test -x`.\n\nPrefer a caller-provided, known-good launcher. Otherwise use this order:\n\n1. `srun --overlap` inside an active Slurm allocation;\n2. `nvshmrun` when available and already configured;\n3. `mpirun` only after a two-rank launcher and hostname preflight succeeds.\n\nNever run a benchmark on a login node. For a same-node run, prove two ranks start on one hostname and use distinct GPUs. For a two-node run, prove one rank starts on each of two distinct hostnames. Do not call a run inter-node without distinct hostname evidence.\n\nCreate a new artifact directory without overwriting prior data:\n\n```text\nnvshmem-performance-\u003CUTC timestamp>\u002F\n  environment\u002F\n  same-node\u002F\n  two-node\u002F\n  commands.md\n  NVSHMEM_PERFORMANCE_REPORT.md\n```\n\nUse additional placement directories for custom suites when needed. Store stdout and stderr separately for every command and record its exit status.\n\n### Collect System Evidence\n\nCollect evidence before benchmarks on every participating compute node. Tell the user that the bundled collector is read-only, then run:\n\n```text\nrun_script(\"scripts\u002Fcollect-performance-environment.sh\")\n```\n\nResolve the script relative to this `SKILL.md`. If `run_script` is unavailable, use `bash scripts\u002Fcollect-performance-environment.sh`. Redirect each node's complete output to `environment\u002F\u003Chostname>.txt`. Preserve partial output even when the collector exits with status 2.\n\nIf direct target access is unavailable, give the user the collector plus the selected launcher preflight and benchmark commands. Ask them to return the complete stdout, stderr, exit statuses, and hostname output. Do not treat login-node evidence as compute-node evidence.\n\n### Run the Initialization Probe\n\nBefore measured tests, run a minimal device-side probe for every placement using the selected launcher and resolved executable:\n\n```bash\nNVSHMEM_INFO=1 \"$PERF_ROOT\u002Fdevice\u002Fpt-to-pt\u002Fshmem_put_latency\" \\\n  -b 4 -e 4 -n 1 -w 1 -t 1 -s thread\n```\n\nPlace the launcher before the executable in the actual command. For example, a Slurm placement uses `NVSHMEM_INFO=1 srun ... \"$PERF_ROOT\u002F...\"`. Capture the information output, selected devices and transport, stdout, stderr, and status. Stop that placement if initialization fails or if rank\u002FGPU placement is wrong.\n\nDo not add `NVSHMEM_DEBUG`, force a transport, select an HCA, disable P2P, or change a tuning variable merely to make the probe pass.\n\n### Run Selected Tests\n\nUse the selected test's standard adjacent `.args` configuration when available. Look beneath the installation's `share\u002Fsrc\u002Fperftest` tree or an explicitly supplied source tree for the matching relative `.args` file.\n\n- For the default suite, use the first nonempty normal line for each test. For `shmem_put_bw`, select the non-`--bidir` line.\n- If the `.args` file has multiple semantic variants, explain only the relevant choices and require a selection instead of running all variants.\n- If no `.args` file is available, inspect the selected executable's `--help` and use its built-in defaults unless the user requested specific limits.\n- Use `--repetitions 3` when supported. If native repetitions are unavailable, run the identical process three times and retain each raw result.\n- Set `NVSHMEM_MACHINE_READABLE_OUTPUT=1`; fall back to the human-readable table when the installed version does not emit machine-readable rows.\n- Run benchmarks serially. Do not overlap tests or silently change message-size ranges, datatypes, atomic operations, scopes, CUDA graph mode, mmap mode, or bidirectionality.\n\nRecord the fully resolved command before execution. Preserve all pre-existing `NVSHMEM_*`, CUDA, provider, launcher, and binding variables in the report.\n\n### Validate Results\n\nValidate every run before summarizing it:\n\n- Require a zero launcher\u002Ftest exit status.\n- Confirm the intended node, PE, and GPU placement from hostname and per-rank output.\n- Confirm the output identifies the selected test or expected metric.\n- Require at least one nonempty message-size or operation row.\n- Reject missing, nonnumeric, `NaN`, infinite, or nonpositive measured values.\n- Preserve the output units; do not silently convert decimal GB\u002Fs to GiB\u002Fs.\n- Confirm three timed repetitions or three separate runs when requested.\n- Search stdout and stderr for initialization, CUDA, bootstrap, transport, timeout, and launch failures.\n\nMark a run `valid`, `invalid`, or `incomplete`. Do not discard partial evidence, average incompatible placements, or call a result expected merely because it is positive.\n\n### Write the Handoff Report\n\nCopy [assets\u002Fperformance-report-template.md](assets\u002Fperformance-report-template.md) to the artifact directory as `NVSHMEM_PERFORMANCE_REPORT.md`, replace every placeholder, and remove unused optional sections.\n\nFor bandwidth tests, report the peak bandwidth and the value at the largest measured message size, then retain the complete curve. For latency tests, report the smallest-message latency and retain the complete curve. Preserve native metrics for atomics, collectives, allocation, and other tests.\n\nUse relative artifact paths in the report so the entire directory can be submitted as one package. Include:\n\n- collection completeness and missing placements;\n- selected tests and rationale;\n- system, software, GPU, NIC, and topology evidence;\n- PE\u002Fnode\u002FGPU binding and `NVSHMEM_INFO` transport evidence;\n- summary and full result tables;\n- exact commands and exit statuses;\n- validation findings, anomalies, assumptions, and raw-log links.\n\nIf no matched baseline is supplied, write `Range assessment: not performed (no matched baseline supplied)`. Do not estimate expected ranges from nominal link speed. If a matched baseline is supplied, compare only identical benchmark, message size, placement, PE count, transport, and relevant options, and state any remaining comparability limits.\n\n## Output Format\n\nReturn the artifact directory and a concise Markdown handoff. Copy the report template to `NVSHMEM_PERFORMANCE_REPORT.md` and include these fields in order:\n\n| Field | Required content |\n| --- | --- |\n| Collection status | `complete`, `incomplete`, or `invalid`, plus unavailable placements and the reason. |\n| Scope and placement | Selected tests, rationale, node count, total PEs, PEs per node, and PE-to-GPU binding. |\n| Environment evidence | Relative paths to per-node collector output, NVSHMEM version, GPU\u002FNIC topology, launcher, and selected transport. |\n| Results | A summary table, complete native-unit result curves, three-repetition evidence, and paths to stdout, stderr, commands, and exit statuses. |\n| Validation and comparison | Per-run validity, anomalies, assumptions, and either a matched-baseline comparison or `Range assessment: not performed (no matched baseline supplied)`. |\n\nUse relative paths so the package remains portable. Do not issue an expected or regressed verdict without a matched baseline.\n\n## Examples\n\nCollect the recommended suite after the user selects it:\n\n```text\nUser: Run the recommended device sanity suite with two PEs on the allocated node.\nAction: Use run_script(\"scripts\u002Fcollect-performance-environment.sh\"), then collect the four device put\u002Fget bandwidth and latency tests with the resolved two-PE placement.\nOutput: Return the artifact-directory path and NVSHMEM_PERFORMANCE_REPORT.md with the same-node results.\n```\n\nStage commands when direct compute-node access is unavailable:\n\n```text\nUser: I need a two-node put-latency baseline but cannot grant you cluster access.\nAction: Request the selected placement and output directory, then provide the collector, launcher preflight, initialization probe, and benchmark commands for the user to run.\nOutput: Package the returned stdout, stderr, exit statuses, and hostname evidence; mark missing data incomplete instead of estimating it.\n```\n\n## Troubleshooting\n\n| Error or condition | Cause | Solution |\n| --- | --- | --- |\n| The collector exits with status 2. | One or more evidence sources are unavailable, but the script produced partial output. | Retain the complete output, record the missing evidence, and rerun on the allocated target compute node if possible. |\n| Initialization probe fails or ranks use the wrong GPUs. | The launcher, bootstrap path, or PE-to-GPU binding is not valid for the placement. | Stop that placement, capture stdout and stderr, correct the known-good launcher or binding, and rerun the probe before measuring. |\n| A selected perftest executable or `.args` file is missing. | The resolved installation or source tree is incomplete. | Report the installation or build problem; do not substitute a different test or invent command-line options. |\n| A result cannot be compared to a baseline. | Benchmark, placement, PE count, transport, or options differ. | Preserve both datasets and report the comparability limit; do not label the result expected or regressed. |\n\n## Limitations\n\n- Treat system evidence as a snapshot of the participating compute nodes and active allocation. It cannot establish site-wide consistency, platform policy, or an untested network path.\n- Assume the standard NVSHMEM perftest set is available; do not inventory it. Treat a selected executable missing at launch as an installation or build problem.\n- Stage commands for the user when direct compute-node access is unavailable. Do not treat login-node evidence as compute-node evidence.\n- Do not install dependencies, allocate nodes, change drivers, modify fabric settings, or edit system configuration as part of collection.\n- Keep collection read-only and unprivileged. Collect only the allowlisted performance and launcher variables emitted by the bundled script; never expose unrelated environment variables or secrets.\n- Never overwrite an existing artifact directory or raw log, replace a failed device test with a host analogue, or represent a same-node result as two-node evidence.\n- Compare results only when the benchmark, message size, placement, PE count, transport, and relevant options match. Without a matched baseline, state that range assessment was not performed and do not issue an expected or regressed verdict.\n",{"data":37,"body":44},{"name":4,"description":6,"license":23,"metadata":38},{"version":39,"author":40,"tags":41},"1.0.0","NVIDIA NVSHMEM Team \u003Cnvshmem@nvidia.com>",[42,14,43],"nvshmem","perftest",{"type":45,"children":46},"root",[47,56,63,69,75,100,106,113,118,151,156,162,199,205,283,289,295,300,368,373,379,384,396,401,412,418,423,456,493,498,534,539,544,553,558,564,569,578,613,618,624,629,750,763,776,782,809,896,909,915,920,971,1000,1006,1026,1031,1036,1081,1094,1100,1112,1223,1228,1234,1239,1248,1253,1262,1268,1374,1380,1418],{"type":48,"tag":49,"props":50,"children":52},"element","h1",{"id":51},"nvshmem-performance-data-collection",[53],{"type":54,"value":55},"text","NVSHMEM Performance Data Collection",{"type":48,"tag":57,"props":58,"children":60},"h2",{"id":59},"purpose",[61],{"type":54,"value":62},"Purpose",{"type":48,"tag":64,"props":65,"children":66},"p",{},[67],{"type":54,"value":68},"Collect reproducible NVSHMEM performance evidence, with the four device put\u002Fget bandwidth and latency tests as the recommended sanity suite. Preserve the system, topology, placement, initialization, transport, commands, raw output, and tuning configuration in a portable Markdown report without inventing system-specific acceptance thresholds.",{"type":48,"tag":57,"props":70,"children":72},{"id":71},"requirements",[73],{"type":54,"value":74},"Requirements",{"type":48,"tag":76,"props":77,"children":78},"ul",{},[79,85,90,95],{"type":48,"tag":80,"props":81,"children":82},"li",{},[83],{"type":54,"value":84},"Use an allocated compute node with the target GPUs visible; never collect evidence or run benchmarks on a login node.",{"type":48,"tag":80,"props":86,"children":87},{},[88],{"type":54,"value":89},"Make the target NVSHMEM installation, selected perftest executable, and a known-good launcher\u002Fbootstrap path available. Direct target access is optional when a user can run the staged commands and return their complete output.",{"type":48,"tag":80,"props":91,"children":92},{},[93],{"type":54,"value":94},"Provide a writable output parent directory for the artifact package. The bundled collector itself is read-only and writes only to stdout.",{"type":48,"tag":80,"props":96,"children":97},{},[98],{"type":54,"value":99},"Require no API keys, privileged access, package installation, or system-configuration changes.",{"type":48,"tag":57,"props":101,"children":103},{"id":102},"inputs",[104],{"type":54,"value":105},"Inputs",{"type":48,"tag":107,"props":108,"children":110},"h3",{"id":109},"required",[111],{"type":54,"value":112},"Required",{"type":48,"tag":64,"props":114,"children":115},{},[116],{"type":54,"value":117},"Resolve these values from the request, the target environment, or focused follow-up before benchmark execution:",{"type":48,"tag":76,"props":119,"children":120},{},[121,126,131,136,141,146],{"type":48,"tag":80,"props":122,"children":123},{},[124],{"type":54,"value":125},"an explicit test, preset, or suite selection;",{"type":48,"tag":80,"props":127,"children":128},{},[129],{"type":54,"value":130},"the NVSHMEM installation prefix and perftest root;",{"type":48,"tag":80,"props":132,"children":133},{},[134],{"type":54,"value":135},"access to the participating compute nodes, or a user who can run and return the staged commands;",{"type":48,"tag":80,"props":137,"children":138},{},[139],{"type":54,"value":140},"a known-good launcher and bootstrap path;",{"type":48,"tag":80,"props":142,"children":143},{},[144],{"type":54,"value":145},"allocated nodes, total PEs, PEs per node, and local PE-to-GPU placement;",{"type":48,"tag":80,"props":147,"children":148},{},[149],{"type":54,"value":150},"the output parent directory for a new artifact package.",{"type":48,"tag":64,"props":152,"children":153},{},[154],{"type":54,"value":155},"For an unspecified request, require only the suite choice first. Do not collect the remaining execution inputs until the user selects a suite. Do not ask again for values that are already explicit or safely observable.",{"type":48,"tag":107,"props":157,"children":159},{"id":158},"optional",[160],{"type":54,"value":161},"Optional",{"type":48,"tag":76,"props":163,"children":164},{},[165,170,184,189,194],{"type":48,"tag":80,"props":166,"children":167},{},[168],{"type":54,"value":169},"exact message-size ranges, datatype, atomic operation, thread-group scope, CUDA graph, mmap, EGM, bidirectional, or other selected-test options;",{"type":48,"tag":80,"props":171,"children":172},{},[173,175,182],{"type":54,"value":174},"an installation or source tree containing the selected test's standard ",{"type":48,"tag":176,"props":177,"children":179},"code",{"className":178},[],[180],{"type":54,"value":181},".args",{"type":54,"value":183}," file;",{"type":48,"tag":80,"props":185,"children":186},{},[187],{"type":54,"value":188},"a caller-provided launcher command, binding policy, artifact-directory name, or reporting notes;",{"type":48,"tag":80,"props":190,"children":191},{},[192],{"type":54,"value":193},"an identical matched baseline for range assessment;",{"type":48,"tag":80,"props":195,"children":196},{},[197],{"type":54,"value":198},"raw stdout, stderr, exit statuses, hostname output, and environment evidence from user-run collection when direct execution is unavailable.",{"type":48,"tag":57,"props":200,"children":202},{"id":201},"scripts",[203],{"type":54,"value":204},"Scripts",{"type":48,"tag":206,"props":207,"children":208},"table",{},[209,233],{"type":48,"tag":210,"props":211,"children":212},"thead",{},[213],{"type":48,"tag":214,"props":215,"children":216},"tr",{},[217,223,228],{"type":48,"tag":218,"props":219,"children":220},"th",{},[221],{"type":54,"value":222},"Script",{"type":48,"tag":218,"props":224,"children":225},{},[226],{"type":54,"value":227},"Use",{"type":48,"tag":218,"props":229,"children":230},{},[231],{"type":54,"value":232},"Invocation",{"type":48,"tag":234,"props":235,"children":236},"tbody",{},[237],{"type":48,"tag":214,"props":238,"children":239},{},[240,250,255],{"type":48,"tag":241,"props":242,"children":243},"td",{},[244],{"type":48,"tag":176,"props":245,"children":247},{"className":246},[],[248],{"type":54,"value":249},"scripts\u002Fcollect-performance-environment.sh",{"type":48,"tag":241,"props":251,"children":252},{},[253],{"type":54,"value":254},"Collect read-only NVSHMEM, GPU, network, RDMA, launcher, topology, and allowlisted environment evidence. It writes to stdout and can exit with status 2 after producing useful partial output.",{"type":48,"tag":241,"props":256,"children":257},{},[258,260,266,268,273,275,281],{"type":54,"value":259},"Resolve the script from this skill directory and use ",{"type":48,"tag":176,"props":261,"children":263},{"className":262},[],[264],{"type":54,"value":265},"run_script",{"type":54,"value":267}," on every participating compute node. If ",{"type":48,"tag":176,"props":269,"children":271},{"className":270},[],[272],{"type":54,"value":265},{"type":54,"value":274}," is unavailable, run ",{"type":48,"tag":176,"props":276,"children":278},{"className":277},[],[279],{"type":54,"value":280},"bash scripts\u002Fcollect-performance-environment.sh",{"type":54,"value":282},"; retain complete output even when it exits nonzero.",{"type":48,"tag":57,"props":284,"children":286},{"id":285},"instructions",[287],{"type":54,"value":288},"Instructions",{"type":48,"tag":107,"props":290,"children":292},{"id":291},"resolve-the-request",[293],{"type":54,"value":294},"Resolve the Request",{"type":48,"tag":64,"props":296,"children":297},{},[298],{"type":54,"value":299},"Classify the request before running a benchmark.",{"type":48,"tag":76,"props":301,"children":302},{},[303,308,313,345,350,355],{"type":48,"tag":80,"props":304,"children":305},{},[306],{"type":54,"value":307},"Treat an exact executable, named family, or named preset suite as an explicit selection.",{"type":48,"tag":80,"props":309,"children":310},{},[311],{"type":54,"value":312},"Treat requests such as \"run perftests,\" \"check NVSHMEM performance,\" or \"perform a sanity check\" as unspecified.",{"type":48,"tag":80,"props":314,"children":315},{},[316,318,324,326],{"type":54,"value":317},"For an unspecified request, read ",{"type":48,"tag":319,"props":320,"children":322},"a",{"href":321},"references\u002Fperftest-catalog.md",[323],{"type":54,"value":321},{"type":54,"value":325},", summarize the available families, recommend the default device sanity suite, and require the user to choose one of:\n",{"type":48,"tag":327,"props":328,"children":329},"ol",{},[330,335,340],{"type":48,"tag":80,"props":331,"children":332},{},[333],{"type":54,"value":334},"the recommended four-test device sanity suite;",{"type":48,"tag":80,"props":336,"children":337},{},[338],{"type":54,"value":339},"a custom device suite;",{"type":48,"tag":80,"props":341,"children":342},{},[343],{"type":54,"value":344},"a host or feature-specific suite.",{"type":48,"tag":80,"props":346,"children":347},{},[348],{"type":54,"value":349},"Run nothing until the user explicitly selects a choice. After a custom category is chosen, ask only for the exact tests, placement, PE count, and special mode that materially affect the run.",{"type":48,"tag":80,"props":351,"children":352},{},[353],{"type":54,"value":354},"If the user already selected a test or suite, skip the menu and honor that scope.",{"type":48,"tag":80,"props":356,"children":357},{},[358,360,366],{"type":54,"value":359},"Use canonical executable names. Do not translate or accept shortened ",{"type":48,"tag":176,"props":361,"children":363},{"className":362},[],[364],{"type":54,"value":365},"_lat",{"type":54,"value":367}," names.",{"type":48,"tag":64,"props":369,"children":370},{},[371],{"type":54,"value":372},"Assume the standard NVSHMEM perftest set exists. Do not inventory executables or enumerate every argument variant before selection. When a selected executable is missing at launch, report an installation or build problem instead of silently substituting another test.",{"type":48,"tag":107,"props":374,"children":376},{"id":375},"default-device-sanity-suite",[377],{"type":54,"value":378},"Default Device Sanity Suite",{"type":48,"tag":64,"props":380,"children":381},{},[382],{"type":54,"value":383},"Use exactly these device point-to-point tests when the user selects the recommended suite:",{"type":48,"tag":385,"props":386,"children":391},"pre",{"className":387,"code":389,"language":54,"meta":390},[388],"language-text","device\u002Fpt-to-pt\u002Fshmem_put_bw\ndevice\u002Fpt-to-pt\u002Fshmem_put_latency\ndevice\u002Fpt-to-pt\u002Fshmem_get_bw\ndevice\u002Fpt-to-pt\u002Fshmem_get_latency\n","",[392],{"type":48,"tag":176,"props":393,"children":394},{"__ignoreMap":390},[395],{"type":54,"value":389},{"type":48,"tag":64,"props":397,"children":398},{},[399],{"type":54,"value":400},"Run each test with exactly two PEs. Measure one same-node placement and, when a two-node allocation is already available, one two-node placement with one PE per node. Do not request or create a new allocation merely to add the two-node placement; record it as unavailable.",{"type":48,"tag":64,"props":402,"children":403},{},[404,406,410],{"type":54,"value":405},"For a selected non-default test, read its entry in ",{"type":48,"tag":319,"props":407,"children":408},{"href":321},[409],{"type":54,"value":321},{"type":54,"value":411},". Use two PEs for point-to-point tests unless the benchmark documents another requirement. Require an explicit node count, PE count, PEs per node, and GPU placement for collective, tile, or allocation tests.",{"type":48,"tag":107,"props":413,"children":415},{"id":414},"prepare-the-target",[416],{"type":54,"value":417},"Prepare the Target",{"type":48,"tag":64,"props":419,"children":420},{},[421],{"type":54,"value":422},"Resolve these values from the user, environment, or direct inspection:",{"type":48,"tag":76,"props":424,"children":425},{},[426,431,436,441,446,451],{"type":48,"tag":80,"props":427,"children":428},{},[429],{"type":54,"value":430},"NVSHMEM installation prefix and perftest root;",{"type":48,"tag":80,"props":432,"children":433},{},[434],{"type":54,"value":435},"launcher and bootstrap path;",{"type":48,"tag":80,"props":437,"children":438},{},[439],{"type":54,"value":440},"allocated nodes and task\u002FGPU capacity;",{"type":48,"tag":80,"props":442,"children":443},{},[444],{"type":54,"value":445},"local PE-to-GPU binding;",{"type":48,"tag":80,"props":447,"children":448},{},[449],{"type":54,"value":450},"selected same-node and two-node placements;",{"type":48,"tag":80,"props":452,"children":453},{},[454],{"type":54,"value":455},"output parent directory.",{"type":48,"tag":64,"props":457,"children":458},{},[459,461,467,469,475,477,483,485,491],{"type":54,"value":460},"Resolve ",{"type":48,"tag":176,"props":462,"children":464},{"className":463},[],[465],{"type":54,"value":466},"PERF_ROOT",{"type":54,"value":468}," from an explicit path first, then ",{"type":48,"tag":176,"props":470,"children":472},{"className":471},[],[473],{"type":54,"value":474},"$NVSHMEM_PREFIX\u002Fbin\u002Fperftest",{"type":54,"value":476},", then ",{"type":48,"tag":176,"props":478,"children":480},{"className":479},[],[481],{"type":54,"value":482},"$NVSHMEM_HOME\u002Fbin\u002Fperftest",{"type":54,"value":484},". Do not search the filesystem for alternate installations. Check only the selected executable paths with ",{"type":48,"tag":176,"props":486,"children":488},{"className":487},[],[489],{"type":54,"value":490},"test -x",{"type":54,"value":492},".",{"type":48,"tag":64,"props":494,"children":495},{},[496],{"type":54,"value":497},"Prefer a caller-provided, known-good launcher. Otherwise use this order:",{"type":48,"tag":327,"props":499,"children":500},{},[501,512,523],{"type":48,"tag":80,"props":502,"children":503},{},[504,510],{"type":48,"tag":176,"props":505,"children":507},{"className":506},[],[508],{"type":54,"value":509},"srun --overlap",{"type":54,"value":511}," inside an active Slurm allocation;",{"type":48,"tag":80,"props":513,"children":514},{},[515,521],{"type":48,"tag":176,"props":516,"children":518},{"className":517},[],[519],{"type":54,"value":520},"nvshmrun",{"type":54,"value":522}," when available and already configured;",{"type":48,"tag":80,"props":524,"children":525},{},[526,532],{"type":48,"tag":176,"props":527,"children":529},{"className":528},[],[530],{"type":54,"value":531},"mpirun",{"type":54,"value":533}," only after a two-rank launcher and hostname preflight succeeds.",{"type":48,"tag":64,"props":535,"children":536},{},[537],{"type":54,"value":538},"Never run a benchmark on a login node. For a same-node run, prove two ranks start on one hostname and use distinct GPUs. For a two-node run, prove one rank starts on each of two distinct hostnames. Do not call a run inter-node without distinct hostname evidence.",{"type":48,"tag":64,"props":540,"children":541},{},[542],{"type":54,"value":543},"Create a new artifact directory without overwriting prior data:",{"type":48,"tag":385,"props":545,"children":548},{"className":546,"code":547,"language":54,"meta":390},[388],"nvshmem-performance-\u003CUTC timestamp>\u002F\n  environment\u002F\n  same-node\u002F\n  two-node\u002F\n  commands.md\n  NVSHMEM_PERFORMANCE_REPORT.md\n",[549],{"type":48,"tag":176,"props":550,"children":551},{"__ignoreMap":390},[552],{"type":54,"value":547},{"type":48,"tag":64,"props":554,"children":555},{},[556],{"type":54,"value":557},"Use additional placement directories for custom suites when needed. Store stdout and stderr separately for every command and record its exit status.",{"type":48,"tag":107,"props":559,"children":561},{"id":560},"collect-system-evidence",[562],{"type":54,"value":563},"Collect System Evidence",{"type":48,"tag":64,"props":565,"children":566},{},[567],{"type":54,"value":568},"Collect evidence before benchmarks on every participating compute node. Tell the user that the bundled collector is read-only, then run:",{"type":48,"tag":385,"props":570,"children":573},{"className":571,"code":572,"language":54,"meta":390},[388],"run_script(\"scripts\u002Fcollect-performance-environment.sh\")\n",[574],{"type":48,"tag":176,"props":575,"children":576},{"__ignoreMap":390},[577],{"type":54,"value":572},{"type":48,"tag":64,"props":579,"children":580},{},[581,583,589,591,596,598,603,605,611],{"type":54,"value":582},"Resolve the script relative to this ",{"type":48,"tag":176,"props":584,"children":586},{"className":585},[],[587],{"type":54,"value":588},"SKILL.md",{"type":54,"value":590},". If ",{"type":48,"tag":176,"props":592,"children":594},{"className":593},[],[595],{"type":54,"value":265},{"type":54,"value":597}," is unavailable, use ",{"type":48,"tag":176,"props":599,"children":601},{"className":600},[],[602],{"type":54,"value":280},{"type":54,"value":604},". Redirect each node's complete output to ",{"type":48,"tag":176,"props":606,"children":608},{"className":607},[],[609],{"type":54,"value":610},"environment\u002F\u003Chostname>.txt",{"type":54,"value":612},". Preserve partial output even when the collector exits with status 2.",{"type":48,"tag":64,"props":614,"children":615},{},[616],{"type":54,"value":617},"If direct target access is unavailable, give the user the collector plus the selected launcher preflight and benchmark commands. Ask them to return the complete stdout, stderr, exit statuses, and hostname output. Do not treat login-node evidence as compute-node evidence.",{"type":48,"tag":107,"props":619,"children":621},{"id":620},"run-the-initialization-probe",[622],{"type":54,"value":623},"Run the Initialization Probe",{"type":48,"tag":64,"props":625,"children":626},{},[627],{"type":54,"value":628},"Before measured tests, run a minimal device-side probe for every placement using the selected launcher and resolved executable:",{"type":48,"tag":385,"props":630,"children":634},{"className":631,"code":632,"language":633,"meta":390,"style":390},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","NVSHMEM_INFO=1 \"$PERF_ROOT\u002Fdevice\u002Fpt-to-pt\u002Fshmem_put_latency\" \\\n  -b 4 -e 4 -n 1 -w 1 -t 1 -s thread\n","bash",[635],{"type":48,"tag":176,"props":636,"children":637},{"__ignoreMap":390},[638,688],{"type":48,"tag":639,"props":640,"children":643},"span",{"class":641,"line":642},"line",1,[644,650,656,662,667,672,677,682],{"type":48,"tag":639,"props":645,"children":647},{"style":646},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[648],{"type":54,"value":649},"NVSHMEM_INFO",{"type":48,"tag":639,"props":651,"children":653},{"style":652},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[654],{"type":54,"value":655},"=",{"type":48,"tag":639,"props":657,"children":659},{"style":658},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[660],{"type":54,"value":661},"1",{"type":48,"tag":639,"props":663,"children":664},{"style":652},[665],{"type":54,"value":666}," \"",{"type":48,"tag":639,"props":668,"children":669},{"style":646},[670],{"type":54,"value":671},"$PERF_ROOT",{"type":48,"tag":639,"props":673,"children":674},{"style":658},[675],{"type":54,"value":676},"\u002Fdevice\u002Fpt-to-pt\u002Fshmem_put_latency",{"type":48,"tag":639,"props":678,"children":679},{"style":652},[680],{"type":54,"value":681},"\"",{"type":48,"tag":639,"props":683,"children":685},{"style":684},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[686],{"type":54,"value":687}," \\\n",{"type":48,"tag":639,"props":689,"children":691},{"class":641,"line":690},2,[692,697,703,708,712,717,722,727,731,736,740,745],{"type":48,"tag":639,"props":693,"children":694},{"style":658},[695],{"type":54,"value":696},"  -b",{"type":48,"tag":639,"props":698,"children":700},{"style":699},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[701],{"type":54,"value":702}," 4",{"type":48,"tag":639,"props":704,"children":705},{"style":658},[706],{"type":54,"value":707}," -e",{"type":48,"tag":639,"props":709,"children":710},{"style":699},[711],{"type":54,"value":702},{"type":48,"tag":639,"props":713,"children":714},{"style":658},[715],{"type":54,"value":716}," -n",{"type":48,"tag":639,"props":718,"children":719},{"style":699},[720],{"type":54,"value":721}," 1",{"type":48,"tag":639,"props":723,"children":724},{"style":658},[725],{"type":54,"value":726}," -w",{"type":48,"tag":639,"props":728,"children":729},{"style":699},[730],{"type":54,"value":721},{"type":48,"tag":639,"props":732,"children":733},{"style":658},[734],{"type":54,"value":735}," -t",{"type":48,"tag":639,"props":737,"children":738},{"style":699},[739],{"type":54,"value":721},{"type":48,"tag":639,"props":741,"children":742},{"style":658},[743],{"type":54,"value":744}," -s",{"type":48,"tag":639,"props":746,"children":747},{"style":658},[748],{"type":54,"value":749}," thread\n",{"type":48,"tag":64,"props":751,"children":752},{},[753,755,761],{"type":54,"value":754},"Place the launcher before the executable in the actual command. For example, a Slurm placement uses ",{"type":48,"tag":176,"props":756,"children":758},{"className":757},[],[759],{"type":54,"value":760},"NVSHMEM_INFO=1 srun ... \"$PERF_ROOT\u002F...\"",{"type":54,"value":762},". Capture the information output, selected devices and transport, stdout, stderr, and status. Stop that placement if initialization fails or if rank\u002FGPU placement is wrong.",{"type":48,"tag":64,"props":764,"children":765},{},[766,768,774],{"type":54,"value":767},"Do not add ",{"type":48,"tag":176,"props":769,"children":771},{"className":770},[],[772],{"type":54,"value":773},"NVSHMEM_DEBUG",{"type":54,"value":775},", force a transport, select an HCA, disable P2P, or change a tuning variable merely to make the probe pass.",{"type":48,"tag":107,"props":777,"children":779},{"id":778},"run-selected-tests",[780],{"type":54,"value":781},"Run Selected Tests",{"type":48,"tag":64,"props":783,"children":784},{},[785,787,792,794,800,802,807],{"type":54,"value":786},"Use the selected test's standard adjacent ",{"type":48,"tag":176,"props":788,"children":790},{"className":789},[],[791],{"type":54,"value":181},{"type":54,"value":793}," configuration when available. Look beneath the installation's ",{"type":48,"tag":176,"props":795,"children":797},{"className":796},[],[798],{"type":54,"value":799},"share\u002Fsrc\u002Fperftest",{"type":54,"value":801}," tree or an explicitly supplied source tree for the matching relative ",{"type":48,"tag":176,"props":803,"children":805},{"className":804},[],[806],{"type":54,"value":181},{"type":54,"value":808}," file.",{"type":48,"tag":76,"props":810,"children":811},{},[812,833,845,865,878,891],{"type":48,"tag":80,"props":813,"children":814},{},[815,817,823,825,831],{"type":54,"value":816},"For the default suite, use the first nonempty normal line for each test. For ",{"type":48,"tag":176,"props":818,"children":820},{"className":819},[],[821],{"type":54,"value":822},"shmem_put_bw",{"type":54,"value":824},", select the non-",{"type":48,"tag":176,"props":826,"children":828},{"className":827},[],[829],{"type":54,"value":830},"--bidir",{"type":54,"value":832}," line.",{"type":48,"tag":80,"props":834,"children":835},{},[836,838,843],{"type":54,"value":837},"If the ",{"type":48,"tag":176,"props":839,"children":841},{"className":840},[],[842],{"type":54,"value":181},{"type":54,"value":844}," file has multiple semantic variants, explain only the relevant choices and require a selection instead of running all variants.",{"type":48,"tag":80,"props":846,"children":847},{},[848,850,855,857,863],{"type":54,"value":849},"If no ",{"type":48,"tag":176,"props":851,"children":853},{"className":852},[],[854],{"type":54,"value":181},{"type":54,"value":856}," file is available, inspect the selected executable's ",{"type":48,"tag":176,"props":858,"children":860},{"className":859},[],[861],{"type":54,"value":862},"--help",{"type":54,"value":864}," and use its built-in defaults unless the user requested specific limits.",{"type":48,"tag":80,"props":866,"children":867},{},[868,870,876],{"type":54,"value":869},"Use ",{"type":48,"tag":176,"props":871,"children":873},{"className":872},[],[874],{"type":54,"value":875},"--repetitions 3",{"type":54,"value":877}," when supported. If native repetitions are unavailable, run the identical process three times and retain each raw result.",{"type":48,"tag":80,"props":879,"children":880},{},[881,883,889],{"type":54,"value":882},"Set ",{"type":48,"tag":176,"props":884,"children":886},{"className":885},[],[887],{"type":54,"value":888},"NVSHMEM_MACHINE_READABLE_OUTPUT=1",{"type":54,"value":890},"; fall back to the human-readable table when the installed version does not emit machine-readable rows.",{"type":48,"tag":80,"props":892,"children":893},{},[894],{"type":54,"value":895},"Run benchmarks serially. Do not overlap tests or silently change message-size ranges, datatypes, atomic operations, scopes, CUDA graph mode, mmap mode, or bidirectionality.",{"type":48,"tag":64,"props":897,"children":898},{},[899,901,907],{"type":54,"value":900},"Record the fully resolved command before execution. Preserve all pre-existing ",{"type":48,"tag":176,"props":902,"children":904},{"className":903},[],[905],{"type":54,"value":906},"NVSHMEM_*",{"type":54,"value":908},", CUDA, provider, launcher, and binding variables in the report.",{"type":48,"tag":107,"props":910,"children":912},{"id":911},"validate-results",[913],{"type":54,"value":914},"Validate Results",{"type":48,"tag":64,"props":916,"children":917},{},[918],{"type":54,"value":919},"Validate every run before summarizing it:",{"type":48,"tag":76,"props":921,"children":922},{},[923,928,933,938,943,956,961,966],{"type":48,"tag":80,"props":924,"children":925},{},[926],{"type":54,"value":927},"Require a zero launcher\u002Ftest exit status.",{"type":48,"tag":80,"props":929,"children":930},{},[931],{"type":54,"value":932},"Confirm the intended node, PE, and GPU placement from hostname and per-rank output.",{"type":48,"tag":80,"props":934,"children":935},{},[936],{"type":54,"value":937},"Confirm the output identifies the selected test or expected metric.",{"type":48,"tag":80,"props":939,"children":940},{},[941],{"type":54,"value":942},"Require at least one nonempty message-size or operation row.",{"type":48,"tag":80,"props":944,"children":945},{},[946,948,954],{"type":54,"value":947},"Reject missing, nonnumeric, ",{"type":48,"tag":176,"props":949,"children":951},{"className":950},[],[952],{"type":54,"value":953},"NaN",{"type":54,"value":955},", infinite, or nonpositive measured values.",{"type":48,"tag":80,"props":957,"children":958},{},[959],{"type":54,"value":960},"Preserve the output units; do not silently convert decimal GB\u002Fs to GiB\u002Fs.",{"type":48,"tag":80,"props":962,"children":963},{},[964],{"type":54,"value":965},"Confirm three timed repetitions or three separate runs when requested.",{"type":48,"tag":80,"props":967,"children":968},{},[969],{"type":54,"value":970},"Search stdout and stderr for initialization, CUDA, bootstrap, transport, timeout, and launch failures.",{"type":48,"tag":64,"props":972,"children":973},{},[974,976,982,984,990,992,998],{"type":54,"value":975},"Mark a run ",{"type":48,"tag":176,"props":977,"children":979},{"className":978},[],[980],{"type":54,"value":981},"valid",{"type":54,"value":983},", ",{"type":48,"tag":176,"props":985,"children":987},{"className":986},[],[988],{"type":54,"value":989},"invalid",{"type":54,"value":991},", or ",{"type":48,"tag":176,"props":993,"children":995},{"className":994},[],[996],{"type":54,"value":997},"incomplete",{"type":54,"value":999},". Do not discard partial evidence, average incompatible placements, or call a result expected merely because it is positive.",{"type":48,"tag":107,"props":1001,"children":1003},{"id":1002},"write-the-handoff-report",[1004],{"type":54,"value":1005},"Write the Handoff Report",{"type":48,"tag":64,"props":1007,"children":1008},{},[1009,1011,1016,1018,1024],{"type":54,"value":1010},"Copy ",{"type":48,"tag":319,"props":1012,"children":1014},{"href":1013},"assets\u002Fperformance-report-template.md",[1015],{"type":54,"value":1013},{"type":54,"value":1017}," to the artifact directory as ",{"type":48,"tag":176,"props":1019,"children":1021},{"className":1020},[],[1022],{"type":54,"value":1023},"NVSHMEM_PERFORMANCE_REPORT.md",{"type":54,"value":1025},", replace every placeholder, and remove unused optional sections.",{"type":48,"tag":64,"props":1027,"children":1028},{},[1029],{"type":54,"value":1030},"For bandwidth tests, report the peak bandwidth and the value at the largest measured message size, then retain the complete curve. For latency tests, report the smallest-message latency and retain the complete curve. Preserve native metrics for atomics, collectives, allocation, and other tests.",{"type":48,"tag":64,"props":1032,"children":1033},{},[1034],{"type":54,"value":1035},"Use relative artifact paths in the report so the entire directory can be submitted as one package. Include:",{"type":48,"tag":76,"props":1037,"children":1038},{},[1039,1044,1049,1054,1066,1071,1076],{"type":48,"tag":80,"props":1040,"children":1041},{},[1042],{"type":54,"value":1043},"collection completeness and missing placements;",{"type":48,"tag":80,"props":1045,"children":1046},{},[1047],{"type":54,"value":1048},"selected tests and rationale;",{"type":48,"tag":80,"props":1050,"children":1051},{},[1052],{"type":54,"value":1053},"system, software, GPU, NIC, and topology evidence;",{"type":48,"tag":80,"props":1055,"children":1056},{},[1057,1059,1064],{"type":54,"value":1058},"PE\u002Fnode\u002FGPU binding and ",{"type":48,"tag":176,"props":1060,"children":1062},{"className":1061},[],[1063],{"type":54,"value":649},{"type":54,"value":1065}," transport evidence;",{"type":48,"tag":80,"props":1067,"children":1068},{},[1069],{"type":54,"value":1070},"summary and full result tables;",{"type":48,"tag":80,"props":1072,"children":1073},{},[1074],{"type":54,"value":1075},"exact commands and exit statuses;",{"type":48,"tag":80,"props":1077,"children":1078},{},[1079],{"type":54,"value":1080},"validation findings, anomalies, assumptions, and raw-log links.",{"type":48,"tag":64,"props":1082,"children":1083},{},[1084,1086,1092],{"type":54,"value":1085},"If no matched baseline is supplied, write ",{"type":48,"tag":176,"props":1087,"children":1089},{"className":1088},[],[1090],{"type":54,"value":1091},"Range assessment: not performed (no matched baseline supplied)",{"type":54,"value":1093},". Do not estimate expected ranges from nominal link speed. If a matched baseline is supplied, compare only identical benchmark, message size, placement, PE count, transport, and relevant options, and state any remaining comparability limits.",{"type":48,"tag":57,"props":1095,"children":1097},{"id":1096},"output-format",[1098],{"type":54,"value":1099},"Output Format",{"type":48,"tag":64,"props":1101,"children":1102},{},[1103,1105,1110],{"type":54,"value":1104},"Return the artifact directory and a concise Markdown handoff. Copy the report template to ",{"type":48,"tag":176,"props":1106,"children":1108},{"className":1107},[],[1109],{"type":54,"value":1023},{"type":54,"value":1111}," and include these fields in order:",{"type":48,"tag":206,"props":1113,"children":1114},{},[1115,1131],{"type":48,"tag":210,"props":1116,"children":1117},{},[1118],{"type":48,"tag":214,"props":1119,"children":1120},{},[1121,1126],{"type":48,"tag":218,"props":1122,"children":1123},{},[1124],{"type":54,"value":1125},"Field",{"type":48,"tag":218,"props":1127,"children":1128},{},[1129],{"type":54,"value":1130},"Required content",{"type":48,"tag":234,"props":1132,"children":1133},{},[1134,1165,1178,1191,1204],{"type":48,"tag":214,"props":1135,"children":1136},{},[1137,1142],{"type":48,"tag":241,"props":1138,"children":1139},{},[1140],{"type":54,"value":1141},"Collection status",{"type":48,"tag":241,"props":1143,"children":1144},{},[1145,1151,1152,1157,1158,1163],{"type":48,"tag":176,"props":1146,"children":1148},{"className":1147},[],[1149],{"type":54,"value":1150},"complete",{"type":54,"value":983},{"type":48,"tag":176,"props":1153,"children":1155},{"className":1154},[],[1156],{"type":54,"value":997},{"type":54,"value":991},{"type":48,"tag":176,"props":1159,"children":1161},{"className":1160},[],[1162],{"type":54,"value":989},{"type":54,"value":1164},", plus unavailable placements and the reason.",{"type":48,"tag":214,"props":1166,"children":1167},{},[1168,1173],{"type":48,"tag":241,"props":1169,"children":1170},{},[1171],{"type":54,"value":1172},"Scope and placement",{"type":48,"tag":241,"props":1174,"children":1175},{},[1176],{"type":54,"value":1177},"Selected tests, rationale, node count, total PEs, PEs per node, and PE-to-GPU binding.",{"type":48,"tag":214,"props":1179,"children":1180},{},[1181,1186],{"type":48,"tag":241,"props":1182,"children":1183},{},[1184],{"type":54,"value":1185},"Environment evidence",{"type":48,"tag":241,"props":1187,"children":1188},{},[1189],{"type":54,"value":1190},"Relative paths to per-node collector output, NVSHMEM version, GPU\u002FNIC topology, launcher, and selected transport.",{"type":48,"tag":214,"props":1192,"children":1193},{},[1194,1199],{"type":48,"tag":241,"props":1195,"children":1196},{},[1197],{"type":54,"value":1198},"Results",{"type":48,"tag":241,"props":1200,"children":1201},{},[1202],{"type":54,"value":1203},"A summary table, complete native-unit result curves, three-repetition evidence, and paths to stdout, stderr, commands, and exit statuses.",{"type":48,"tag":214,"props":1205,"children":1206},{},[1207,1212],{"type":48,"tag":241,"props":1208,"children":1209},{},[1210],{"type":54,"value":1211},"Validation and comparison",{"type":48,"tag":241,"props":1213,"children":1214},{},[1215,1217,1222],{"type":54,"value":1216},"Per-run validity, anomalies, assumptions, and either a matched-baseline comparison or ",{"type":48,"tag":176,"props":1218,"children":1220},{"className":1219},[],[1221],{"type":54,"value":1091},{"type":54,"value":492},{"type":48,"tag":64,"props":1224,"children":1225},{},[1226],{"type":54,"value":1227},"Use relative paths so the package remains portable. Do not issue an expected or regressed verdict without a matched baseline.",{"type":48,"tag":57,"props":1229,"children":1231},{"id":1230},"examples",[1232],{"type":54,"value":1233},"Examples",{"type":48,"tag":64,"props":1235,"children":1236},{},[1237],{"type":54,"value":1238},"Collect the recommended suite after the user selects it:",{"type":48,"tag":385,"props":1240,"children":1243},{"className":1241,"code":1242,"language":54,"meta":390},[388],"User: Run the recommended device sanity suite with two PEs on the allocated node.\nAction: Use run_script(\"scripts\u002Fcollect-performance-environment.sh\"), then collect the four device put\u002Fget bandwidth and latency tests with the resolved two-PE placement.\nOutput: Return the artifact-directory path and NVSHMEM_PERFORMANCE_REPORT.md with the same-node results.\n",[1244],{"type":48,"tag":176,"props":1245,"children":1246},{"__ignoreMap":390},[1247],{"type":54,"value":1242},{"type":48,"tag":64,"props":1249,"children":1250},{},[1251],{"type":54,"value":1252},"Stage commands when direct compute-node access is unavailable:",{"type":48,"tag":385,"props":1254,"children":1257},{"className":1255,"code":1256,"language":54,"meta":390},[388],"User: I need a two-node put-latency baseline but cannot grant you cluster access.\nAction: Request the selected placement and output directory, then provide the collector, launcher preflight, initialization probe, and benchmark commands for the user to run.\nOutput: Package the returned stdout, stderr, exit statuses, and hostname evidence; mark missing data incomplete instead of estimating it.\n",[1258],{"type":48,"tag":176,"props":1259,"children":1260},{"__ignoreMap":390},[1261],{"type":54,"value":1256},{"type":48,"tag":57,"props":1263,"children":1265},{"id":1264},"troubleshooting",[1266],{"type":54,"value":1267},"Troubleshooting",{"type":48,"tag":206,"props":1269,"children":1270},{},[1271,1292],{"type":48,"tag":210,"props":1272,"children":1273},{},[1274],{"type":48,"tag":214,"props":1275,"children":1276},{},[1277,1282,1287],{"type":48,"tag":218,"props":1278,"children":1279},{},[1280],{"type":54,"value":1281},"Error or condition",{"type":48,"tag":218,"props":1283,"children":1284},{},[1285],{"type":54,"value":1286},"Cause",{"type":48,"tag":218,"props":1288,"children":1289},{},[1290],{"type":54,"value":1291},"Solution",{"type":48,"tag":234,"props":1293,"children":1294},{},[1295,1313,1331,1356],{"type":48,"tag":214,"props":1296,"children":1297},{},[1298,1303,1308],{"type":48,"tag":241,"props":1299,"children":1300},{},[1301],{"type":54,"value":1302},"The collector exits with status 2.",{"type":48,"tag":241,"props":1304,"children":1305},{},[1306],{"type":54,"value":1307},"One or more evidence sources are unavailable, but the script produced partial output.",{"type":48,"tag":241,"props":1309,"children":1310},{},[1311],{"type":54,"value":1312},"Retain the complete output, record the missing evidence, and rerun on the allocated target compute node if possible.",{"type":48,"tag":214,"props":1314,"children":1315},{},[1316,1321,1326],{"type":48,"tag":241,"props":1317,"children":1318},{},[1319],{"type":54,"value":1320},"Initialization probe fails or ranks use the wrong GPUs.",{"type":48,"tag":241,"props":1322,"children":1323},{},[1324],{"type":54,"value":1325},"The launcher, bootstrap path, or PE-to-GPU binding is not valid for the placement.",{"type":48,"tag":241,"props":1327,"children":1328},{},[1329],{"type":54,"value":1330},"Stop that placement, capture stdout and stderr, correct the known-good launcher or binding, and rerun the probe before measuring.",{"type":48,"tag":214,"props":1332,"children":1333},{},[1334,1346,1351],{"type":48,"tag":241,"props":1335,"children":1336},{},[1337,1339,1344],{"type":54,"value":1338},"A selected perftest executable or ",{"type":48,"tag":176,"props":1340,"children":1342},{"className":1341},[],[1343],{"type":54,"value":181},{"type":54,"value":1345}," file is missing.",{"type":48,"tag":241,"props":1347,"children":1348},{},[1349],{"type":54,"value":1350},"The resolved installation or source tree is incomplete.",{"type":48,"tag":241,"props":1352,"children":1353},{},[1354],{"type":54,"value":1355},"Report the installation or build problem; do not substitute a different test or invent command-line options.",{"type":48,"tag":214,"props":1357,"children":1358},{},[1359,1364,1369],{"type":48,"tag":241,"props":1360,"children":1361},{},[1362],{"type":54,"value":1363},"A result cannot be compared to a baseline.",{"type":48,"tag":241,"props":1365,"children":1366},{},[1367],{"type":54,"value":1368},"Benchmark, placement, PE count, transport, or options differ.",{"type":48,"tag":241,"props":1370,"children":1371},{},[1372],{"type":54,"value":1373},"Preserve both datasets and report the comparability limit; do not label the result expected or regressed.",{"type":48,"tag":57,"props":1375,"children":1377},{"id":1376},"limitations",[1378],{"type":54,"value":1379},"Limitations",{"type":48,"tag":76,"props":1381,"children":1382},{},[1383,1388,1393,1398,1403,1408,1413],{"type":48,"tag":80,"props":1384,"children":1385},{},[1386],{"type":54,"value":1387},"Treat system evidence as a snapshot of the participating compute nodes and active allocation. It cannot establish site-wide consistency, platform policy, or an untested network path.",{"type":48,"tag":80,"props":1389,"children":1390},{},[1391],{"type":54,"value":1392},"Assume the standard NVSHMEM perftest set is available; do not inventory it. Treat a selected executable missing at launch as an installation or build problem.",{"type":48,"tag":80,"props":1394,"children":1395},{},[1396],{"type":54,"value":1397},"Stage commands for the user when direct compute-node access is unavailable. Do not treat login-node evidence as compute-node evidence.",{"type":48,"tag":80,"props":1399,"children":1400},{},[1401],{"type":54,"value":1402},"Do not install dependencies, allocate nodes, change drivers, modify fabric settings, or edit system configuration as part of collection.",{"type":48,"tag":80,"props":1404,"children":1405},{},[1406],{"type":54,"value":1407},"Keep collection read-only and unprivileged. Collect only the allowlisted performance and launcher variables emitted by the bundled script; never expose unrelated environment variables or secrets.",{"type":48,"tag":80,"props":1409,"children":1410},{},[1411],{"type":54,"value":1412},"Never overwrite an existing artifact directory or raw log, replace a failed device test with a host analogue, or represent a same-node result as two-node evidence.",{"type":48,"tag":80,"props":1414,"children":1415},{},[1416],{"type":54,"value":1417},"Compare results only when the benchmark, message size, placement, PE count, transport, and relevant options match. Without a matched baseline, state that range assessment was not performed and do not issue an expected or regressed verdict.",{"type":48,"tag":1419,"props":1420,"children":1421},"style",{},[1422],{"type":54,"value":1423},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1425,"total":1506},[1426,1432,1446,1460,1472,1484,1496],{"slug":4,"name":4,"fn":5,"description":6,"org":1427,"tags":1428,"stars":20,"repoUrl":21,"updatedAt":22},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1429,1430,1431],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":1433,"name":1433,"fn":1434,"description":1435,"org":1436,"tags":1437,"stars":20,"repoUrl":21,"updatedAt":1445},"nvshmem-configure-nic-pe-mapping","configure NVSHMEM NIC-to-PE mappings","Recommend NVSHMEM NIC-to-PE mappings and environment exports. Use for HCA selection, multi-NIC configuration, or topology-based mapping diagnostics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1438,1441,1444],{"name":1439,"slug":1440,"type":15},"Configuration","configuration",{"name":1442,"slug":1443,"type":15},"Networking","networking",{"name":9,"slug":8,"type":15},"2026-08-31T09:19:06.10294",{"slug":1447,"name":1447,"fn":1448,"description":1449,"org":1450,"tags":1451,"stars":20,"repoUrl":21,"updatedAt":1459},"nvshmem-docs","fetch official NVSHMEM documentation","Find version-aware official NVSHMEM and NVSHMEM4Py documentation for releases, installation, APIs, runtime settings, transports, containers, and troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1452,1455,1456],{"name":1453,"slug":1454,"type":15},"Documentation","documentation",{"name":9,"slug":8,"type":15},{"name":1457,"slug":1458,"type":15},"Reference","reference","2026-08-31T09:19:00.214841",{"slug":1461,"name":1461,"fn":1462,"description":1463,"org":1464,"tags":1465,"stars":20,"repoUrl":21,"updatedAt":1471},"nvshmem-enable-tma","optimize NVSHMEM CUDA kernels for TMA","Prepare or review NVSHMEM CUDA kernels for TMA SMEM registration and direct-SMEM transfers. Do not use for unrelated CUDA tuning.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1466,1469,1470],{"name":1467,"slug":1468,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-08-31T09:18:50.449676",{"slug":1473,"name":1473,"fn":1474,"description":1475,"org":1476,"tags":1477,"stars":20,"repoUrl":21,"updatedAt":1483},"nvshmem-get-started","onboard developers to NVSHMEM","Guide NVSHMEM beginners through fit assessment, mental models, first C\u002FC++ or Python NVSHMEM programs, compilation, launching, and next steps. Use for onboarding.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1478,1479,1480],{"name":1453,"slug":1454,"type":15},{"name":9,"slug":8,"type":15},{"name":1481,"slug":1482,"type":15},"Onboarding","onboarding","2026-08-31T09:18:55.48573",{"slug":1485,"name":1485,"fn":1486,"description":1487,"org":1488,"tags":1489,"stars":20,"repoUrl":21,"updatedAt":1495},"nvshmem-install","install and validate NVSHMEM","Plan and validate NVSHMEM and NVSHMEM4Py installations. Use for package, container, or source deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1490,1493,1494],{"name":1491,"slug":1492,"type":15},"Deployment","deployment",{"name":1467,"slug":1468,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:18:50.105521",{"slug":1497,"name":1497,"fn":1498,"description":1499,"org":1500,"tags":1501,"stars":20,"repoUrl":21,"updatedAt":1505},"nvshmem-select-remote-transport","select NVSHMEM remote transport configurations","Select an NVSHMEM remote transport from target system and kernel evidence. Use for inter-node selection, compatibility checks, or configuration.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1502,1503,1504],{"name":1439,"slug":1440,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-08-31T09:18:59.493722",9,{"items":1508,"total":1662},[1509,1525,1540,1551,1563,1577,1590,1604,1617,1628,1642,1651],{"slug":1510,"name":1510,"fn":1511,"description":1512,"org":1513,"tags":1514,"stars":1522,"repoUrl":1523,"updatedAt":1524},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1515,1516,1519],{"name":1453,"slug":1454,"type":15},{"name":1517,"slug":1518,"type":15},"MCP","mcp",{"name":1520,"slug":1521,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":1526,"name":1526,"fn":1527,"description":1528,"org":1529,"tags":1530,"stars":1537,"repoUrl":1538,"updatedAt":1539},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1531,1534,1535],{"name":1532,"slug":1533,"type":15},"Containers","containers",{"name":1491,"slug":1492,"type":15},{"name":1536,"slug":30,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1541,"name":1541,"fn":1542,"description":1543,"org":1544,"tags":1545,"stars":1537,"repoUrl":1538,"updatedAt":1550},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1546,1549],{"name":1547,"slug":1548,"type":15},"CI\u002FCD","ci-cd",{"name":1491,"slug":1492,"type":15},"2026-07-14T05:25:59.97109",{"slug":1552,"name":1552,"fn":1553,"description":1554,"org":1555,"tags":1556,"stars":1537,"repoUrl":1538,"updatedAt":1562},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1557,1558,1559],{"name":1547,"slug":1548,"type":15},{"name":1491,"slug":1492,"type":15},{"name":1560,"slug":1561,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":1564,"name":1564,"fn":1565,"description":1566,"org":1567,"tags":1568,"stars":1537,"repoUrl":1538,"updatedAt":1576},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1569,1572,1573],{"name":1570,"slug":1571,"type":15},"Debugging","debugging",{"name":1560,"slug":1561,"type":15},{"name":1574,"slug":1575,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1578,"name":1578,"fn":1579,"description":1580,"org":1581,"tags":1582,"stars":1537,"repoUrl":1538,"updatedAt":1589},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1583,1586],{"name":1584,"slug":1585,"type":15},"Best Practices","best-practices",{"name":1587,"slug":1588,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1591,"name":1591,"fn":1592,"description":1593,"org":1594,"tags":1595,"stars":1537,"repoUrl":1538,"updatedAt":1603},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1596,1599,1602],{"name":1597,"slug":1598,"type":15},"Machine Learning","machine-learning",{"name":1600,"slug":1601,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":1605,"name":1605,"fn":1606,"description":1607,"org":1608,"tags":1609,"stars":1537,"repoUrl":1538,"updatedAt":1616},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1610,1613],{"name":1611,"slug":1612,"type":15},"QA","qa",{"name":1614,"slug":1615,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1618,"name":1618,"fn":1619,"description":1620,"org":1621,"tags":1622,"stars":1537,"repoUrl":1538,"updatedAt":1627},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1623,1624],{"name":1491,"slug":1492,"type":15},{"name":1625,"slug":1626,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1629,"name":1629,"fn":1630,"description":1631,"org":1632,"tags":1633,"stars":1537,"repoUrl":1538,"updatedAt":1641},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1634,1637,1638],{"name":1635,"slug":1636,"type":15},"Code Review","code-review",{"name":1560,"slug":1561,"type":15},{"name":1639,"slug":1640,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":1643,"name":1643,"fn":1644,"description":1645,"org":1646,"tags":1647,"stars":1537,"repoUrl":1538,"updatedAt":1650},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1648,1649],{"name":1611,"slug":1612,"type":15},{"name":1614,"slug":1615,"type":15},"2026-07-14T05:25:54.928983",{"slug":1652,"name":1652,"fn":1653,"description":1654,"org":1655,"tags":1656,"stars":1537,"repoUrl":1538,"updatedAt":1661},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1657,1660],{"name":1658,"slug":1659,"type":15},"Automation","automation",{"name":1547,"slug":1548,"type":15},"2026-07-30T05:29:03.275638",563]