[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-fault-injection-loop":3,"mdc-fvs3w9-key":34,"related-org-nvidia-fault-injection-loop":3847,"related-repo-nvidia-fault-injection-loop":4006},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"fault-injection-loop","run closed-loop fault injection benchmarks","Closed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs \u002Flog-analysis and \u002Ffr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close them.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Benchmarking","benchmarking","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Simulation","simulation",{"name":21,"slug":22,"type":15},"Testing","testing",308,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-resiliency-ext","2026-07-14T05:35:12.409884",null,56,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the effective training time by minimizing the downtime due to failures and interruptions.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-resiliency-ext\u002Ftree\u002FHEAD\u002Fsrc\u002Fnvidia_resiliency_ext\u002Fskills\u002Fnvrx-attr\u002Ffault-injection-loop","---\nname: fault-injection-loop\ndescription: >\n  Closed-loop fault injection and attribution accuracy benchmark. Draws from a\n  prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them\n  2 at a time via sbatch — waiting for each pair to finish before submitting the\n  next — to bound filesystem load. GPU-related faults are front-loaded in the pool.\n  After all jobs complete, runs \u002Flog-analysis and \u002Ffr-analysis on every experiment,\n  scores attribution vs. ground truth, aggregates gaps, and iterates on attribution\n  modules to close them.\ncompatibility: Requires SLURM cluster access, sbatch, LLM_API_KEY, langchain-openai, logsage, and nvidia-resiliency-ext installed. This workflow has only been validated with Megatron-LM workloads.\nmetadata:\n  author: nvidia\n  sub-skills: [log-analysis, fr-analysis]\n---\n\n# Skill: fault-injection-loop\n\nIterative closed-loop skill that runs a prioritized fault-injection experiment pool\n2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates\ngaps across the matrix, and proposes targeted improvements to attribution modules.\n\n---\n\n## Overview\n\n```\n┌───────────────────────────────────────────────────────────────────────┐\n│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │\n│               GPU faults first, then crash, Python-hang, signal       │\n│                                                                        │\n│  repeat until pool exhausted:                                          │\n│  1. SUBMIT   → sbatch 2 jobs from pool head                            │\n│  2. WAIT     → poll until both jobs leave RUNNING\u002FPENDING              │\n│                                                                        │\n│  after all jobs done:                                                  │\n│  3. ANALYZE  → watch_and_analyze.sh: \u002Flog-analysis + \u002Ffr-analysis     │\n│               per completed job, streaming as jobs finish              │\n│  4. SCORE    → compare attribution output vs injected ground truth     │\n│  5. AGGREGATE→ build results table; identify systematic failure modes  │\n│  6. IMPROVE  → patch log_analyzer\u002Fnvrx_logsage.py                     │\n│  7. LOOP     → re-run same pool with updated attribution code          │\n└───────────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Step 0 — Fault Pool Design\n\nThe pool is defined as an ordered list of `FAULT_TYPE:RANK:ITER:NODES` entries\ninside `scripts\u002Fprepare_node_alloc.sh`. Default pool (34 experiments, 17 batches):\n\n```\n# GPU hangs — highest priority; full rank sweep across all node counts\nGPU_SLEEP:1:5:2   GPU_SLEEP:0:5:2      # 2-node: rank-1, rank-0\nGPU_SLEEP:4:5:2   GPU_SLEEP:7:5:2      # 2-node: mid-rank, last-rank\nGPU_SLEEP:1:5:4   GPU_SLEEP:0:5:4      # 4-node: rank-1, rank-0\nGPU_SLEEP:8:5:4   GPU_SLEEP:15:5:4     # 4-node: mid, last\nGPU_SLEEP:1:5:8   GPU_SLEEP:0:5:8      # 8-node: rank-1, rank-0\nGPU_SLEEP:16:5:8  GPU_SLEEP:31:5:8     # 8-node: mid, last\n\n# GPU errors — high priority; rank-0 and rank-1 across all node counts\nGPU_ERROR:1:5:2   GPU_ERROR:0:5:2\nGPU_ERROR:1:5:4   GPU_ERROR:0:5:4\nGPU_ERROR:1:5:8   GPU_ERROR:0:5:8\n\n# Crash faults\nSIGKILL:1:5:2     SIGKILL:0:5:2\nSIGKILL:1:5:4     SIGKILL:1:5:8\nSEGFAULT:1:5:2    SEGFAULT:0:5:2\nSEGFAULT:1:5:4    OS_ABORT:1:5:2\n\n# Python-level hangs\nLOCK_GIL:1:5:2    LOCK_GIL:0:5:2\nWORKLOAD_EXC:1:5:2  ASYNC_EXC:1:5:2\n\n# Signals\nSIGTERM:1:5:2     SIGINT:1:5:2\nSIGSTOP:1:5:2     SIGNAL_EXC:1:5:2\n```\n\nRank coverage per node count (4 GPUs\u002Fnode):\n\n| Nodes | Total ranks | rank-0 | rank-1 | mid | last |\n|-------|-------------|--------|--------|-----|------|\n| 2     | 8           | 0      | 1      | 4   | 7    |\n| 4     | 16          | 0      | 1      | 8   | 15   |\n| 8     | 32          | 0      | 1      | 16  | 31   |\n\nTo run a custom subset, override `POOL` before calling the script:\n```bash\nPOOL=\"GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2\" bash scripts\u002Fprepare_node_alloc.sh\n```\n\n## Local User Config\n\nStart from the tracked template:\n\n```bash\ncp scripts\u002Fuser.env.example scripts\u002Fuser.env\n```\n\nThen edit `scripts\u002Fuser.env` with cluster-specific settings. This file is\nsourced by `run_session.sh`, `prepare_node_alloc.sh`, `watch_and_analyze.sh`,\n`l4_gb200_reduced.sh`, and `n3_super_gb200_fi.sh`. It is required for this skill\nto run and is intended to stay local and untracked.\n\nRecommended contents:\n\n```bash\nPARTITION=gb-nvl-134-135\nBASE_EXPERIMENTS_DIR=\"${HOME}\u002Fnvrx-attr-experiments\"\nMEGATRON_REPO_HOST_PATH=\"${HOME}\u002Fmegatron-lm-main\"\nSHARED_TMP_BASE_DIR=\"${HOME}\u002Ftmp\"\nWORKSPACE_HOST_PATH=\"${HOME}\u002Ftmp\"\nCONTAINER_IMAGE=\"nvcr.io\u002Fnvidia\u002Fnemo:26.04\"\nLLM_API_KEY_FILE=\"${HOME}\u002F.llm_api_key\"\nJUDGE_API_KEY_FILE=\"${HOME}\u002F.llm_api_key\"\nNVRX_LLM_MODEL=\"nvidia\u002Fnemotron-3-super-120b-a12b\"\nNVRX_LLM_BASE_URL=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\"\nJUDGE_MODEL=\"qwen\u002Fqwen3.5-397b-a17b\"\nJUDGE_BASE_URL=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\"\nFR_SEGMENT_SIZE=32\n```\n\nUse `user.env` for stable site defaults such as partition, container image, and\nhost paths, plus local LLM credentials and endpoint settings for log-analysis\nand the judge. Use per-run environment overrides for experiment-specific\ncontrols such as `POOL`, `WORKLOAD`, `BATCH_SIZE`, `FAULT_TYPE`,\n`FAULT_AT_ITER`, or `FAULT_DELAY`.\n\nIf you use local Triton\u002FInductor cache staging, set the cache variables in\n`scripts\u002Fuser.env`. See `scripts\u002Fuser.env.example` for the supported\n`ENABLE_NFS_CACHE_STAGING`, `NFS_TRITON_CACHE`, and `NFS_INDUCTOR_CACHE`\nentries and workload-specific path examples.\n\nEnvironment variables:\n\n| Variable | Default | Description |\n|---|---|---|\n| `WORKLOAD` | `llama4_scout` | Select a registered workload by name (see `scripts\u002Fworkloads.conf`) |\n| `ACCOUNT` | _(cluster default or `scripts\u002Fuser.env`)_ | SLURM account |\n| `PARTITION` | _(cluster default or `scripts\u002Fuser.env`)_ | SLURM partition |\n| `GPUS_PER_NODE` | `4` | GPUs per node |\n| `TIME` | `00:30:00` | Per-job wall-clock limit |\n| `BATCH_SIZE` | `2` | Jobs submitted per round |\n| `POLL_INTERVAL` | `30` | Seconds between queue polls |\n| `BASE_EXPERIMENTS_DIR` | `${HOME}\u002Fnvrx-attr-experiments` | Root for all output |\n| `MEGATRON_REPO_HOST_PATH` | `${HOME}\u002Fmegatron-lm-main` | Host path to the Megatron checkout mounted into the container |\n| `SHARED_TMP_BASE_DIR` | `${HOME}\u002Ftmp` | Shared filesystem path used for cross-step coordination |\n| `WORKSPACE_HOST_PATH` | `${HOME}\u002Ftmp` | Host path mounted at `\u002Fworkspace` inside the container |\n| `CONTAINER_IMAGE` | `nvcr.io\u002Fnvidia\u002Fnemo:26.04` | Container image used by the workload script |\n| `LLM_API_KEY_FILE` | _unset_ | File containing the log-analysis API key |\n| `JUDGE_API_KEY_FILE` | _unset_ | File containing the judge API key |\n| `NVRX_LLM_MODEL` | `nvidia\u002Fnemotron-3-super-120b-a12b` | Model for log-analysis |\n| `NVRX_LLM_BASE_URL` | `https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1` | Base URL for log-analysis |\n| `JUDGE_MODEL` | `qwen\u002Fqwen3.5-397b-a17b` | Model for judge scoring |\n| `JUDGE_BASE_URL` | `https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1` | Base URL for judge scoring |\n| `FR_SEGMENT_SIZE` | `32` | Ranks per segment for coarse FR scoring |\n| `SBATCH_SCRIPT` | `scripts\u002Fl4_gb200_reduced.sh` | Job script to submit |\n| `POOL` | _(default pool above)_ | Space-separated experiment triplets |\n\n### Registered workloads (`scripts\u002Fworkloads.conf`)\n\n| Name | Script | Base dir | Description |\n|---|---|---|---|\n| `llama4_scout` | `l4_gb200_reduced.sh` | `${HOME}\u002Fnvrx-attr-experiments` | Llama4-Scout (reduced layers) on GB200; minimum supported size is 2 nodes |\n| `n3_super` | `n3_super_gb200_fi.sh` | `${HOME}\u002Fnvrx-attr-experiments` | Nemotron3-Super on GB200; minimum supported size is 8 nodes |\n\nWorkload note:\n- `llama4_scout` requires at least 2 nodes.\n- `n3_super` requires at least 8 nodes. Its default registered pool contains only 8-node experiments.\n\n```bash\n# Run the full pool against the validated example workload\nbash scripts\u002Fprepare_node_alloc.sh\n\n# Run a custom subset against llama4_scout\nPOOL=\"GPU_SLEEP:1:5:2 SIGKILL:1:5:2\" WORKLOAD=llama4_scout bash scripts\u002Fprepare_node_alloc.sh\n```\n\n---\n\n## Step 1 & 2 — Batched Submission + Wait (automated)\n\n```bash\nbash scripts\u002Fprepare_node_alloc.sh\n```\n\nThe script loops: submit 2 jobs → poll `squeue` every 30 s until both finish →\nsubmit next 2. Progress is printed inline:\n\n```\n>>> Batch 1: experiments 1–2 of 34\n  submitted: GPU_SLEEP rank=1  iter=5 nodes=2 -> job=1850\n  submitted: GPU_SLEEP rank=0  iter=5 nodes=2 -> job=1851\n  waiting for GPU_SLEEP:1:5:2 GPU_SLEEP:0:5:2 (1850,1851) ... 30s 60s done.\n>>> Batch 2: experiments 3–4 of 34\n  ...\n```\n\nA session directory and TSV tracking file are created at launch time:\n```\n${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002F\u003CYYYYMMDD_HHMMSS>\u002F\n  experiments.tsv                              ← tracking file (all job IDs + paths)\n  n\u003CN>_\u003CFAULT>_r\u003CR>_i\u003CI>\u002F                     ← one subdir per experiment\n    logs\u002Fslurm\u002F\u003CJOB_ID>.launch.out\n    logs\u002Fslurm\u002F\u003CJOB_ID>.*.1.main_workload.log  ← log-analysis input\n    checkpoints\u002F                               ← fr-analysis input (FR dumps)\n    tensorboard\u002F\n  experiments_report.md                        ← generated by watch_and_analyze.sh\n```\n\nTracking file columns: `JOB_ID  FAULT_TYPE  RANK  ITER  NODES  EXPERIMENT_DIR`\n\n---\n\n## Step 3 — Analyze All Experiments\n\nRun the watcher\u002Fanalyzer — it reads the tracking file and processes each experiment\nas its job state leaves RUNNING\u002FPENDING (works whether jobs are still running or\nalready done):\n\n```bash\nbash scripts\u002Fwatch_and_analyze.sh \\\n    ${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002F\u003CYYYYMMDD_HHMMSS>\u002Fexperiments.tsv\n```\n\nThe watcher:\n1. Reads each row from the tracking TSV\n2. Calls `nvrx_logsage.py --exclude_nvrx_logs` and parses the text output to get\n   `restart_decision` and `attribution_text`\n3. Calls FR analysis as `python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path \"${EXPERIMENT_DIR}\u002Fcheckpoints\" -p \"_dump_*\"` and passes the raw table output to the judge\n4. Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)\n5. Appends a scored row to `\u003Csession>_report.md` as a markdown table row\n6. Repeats until all experiments are analyzed\n\nTo also run the sub-skills interactively for a single experiment:\n```bash\n\u002Flog-analysis --log-path \"${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F${JOB_ID}.*.1.main_workload.log\"\n\u002Ffr-analysis  --fr-path \"${EXPERIMENT_DIR}\u002Fcheckpoints\" -p \"_dump_*\"\n```\n\n---\n\n## Step 4 — Score Each Experiment\n\nScoring is performed by `scripts\u002Fscore_attribution.py`, an LLM judge that\nreceives the ground truth, the filtered raw log, the logsage attribution output, and the FR\nanalysis output, then returns structured JSON scores with a reasoning note.\n\n| Column | Values | Meaning |\n|---|---|---|\n| **restart_correct** | `true` \u002F `false` \u002F `N\u002FA` | Restart decision matches expected for this fault type |\n| **rank_primary** | `true` \u002F `false` \u002F `partial` | Injected rank is the primary root-cause in attribution |\n| **rank_any** | `true` \u002F `false` | Injected rank mentioned anywhere in attribution |\n| **fault_described** | `true` \u002F `false` \u002F `partial` | Fault nature (hang\u002Fcrash\u002Fsignal\u002Fexception) correctly described |\n| **fr_rank_correct** | `rank` \u002F `node` \u002F `segment` \u002F `false` \u002F `no_dumps` | FR analysis narrows correctly to the injected rank, exactly one `GPUS_PER_NODE` rank block containing that rank, the configured `FR_SEGMENT_SIZE` rank block containing the injected rank, or fails to narrow usefully |\n| **judge_notes** | string | One-sentence summary of the main gap or confirmation |\n\nThe judge is given:\n1. Ground truth: `fault_type`, `rank`, `iter`, `nodes`\n2. Expected restart decision + rationale (derived from `score_attribution.py:_RESTART_TABLE`)\n3. Filtered raw log (last 400 lines, same `exclude_nvrx_logs` filtering as logsage)\n4. Raw logsage stdout (5-field text format)\n5. Raw FR analysis table output from `fr_attribution.py --fr-path ... -p \"_dump_*\"`\n6. `GPUS_PER_NODE` and `FR_SEGMENT_SIZE` to map the injected rank to exact node-sized and segment-sized scopes for FR scoring\n\nDefault judge model: `qwen\u002Fqwen3.5-397b-a17b`. Override with `--model` in `score_attribution.py`.\nDefault segment size for FR scope scoring: `32` ranks. Override with `FR_SEGMENT_SIZE`.\n\n---\n\n## Step 5 — Aggregate Results\n\nThe canonical output of the loop is the markdown table in `\u003Csession>_report.md`.\nWhen summarizing results for users, prefer linking to that file and reproducing the\nsame table shape rather than flattening the results into plain prose.\n\nThe report markdown table from `watch_and_analyze.sh` gives a matrix view. Look for\npatterns across rows:\n\n```\n| FAULT_TYPE | NODES | RANK | restart_correct | rank_primary | rank_any | fault_described | fr_rank_correct | judge_notes |\n|------------|-------|------|-----------------|--------------|----------|-----------------|-----------------|-------------|\n| GPU_SLEEP  |   2   |  0   |      true       |    false     |   true   |      true       |      true       | rank-0 identified only in secondary issues |\n| GPU_SLEEP  |   2   |  1   |      true       |     true     |   true   |      true       |      true       | correct on all dimensions |\n| GPU_ERROR  |   2   |  1   |      false      |    false     |  false   |     partial     |      true       | LLM issued RESTART; rank not mentioned |\n| SIGKILL    |   2   |  0   |      true       |    false     |  false   |     false       |      true       | attribution describes timeout not kill signal |\n```\n\nCommon failure mode patterns and their meaning:\n\n| Pattern | Interpretation |\n|---|---|\n| `rank_primary=false`, `rank_any=true` | Rank detected but treated as collateral; logsage putting it in secondary issues |\n| `rank_any=false` for rank-0 | Rank-0 hang silences watchdog on other ranks; logsage lacks rank-0 signal |\n| `fault_described=partial` for crash types | Crash keywords present but fault type not specifically named |\n| `restart_correct=false` for GPU_ERROR | LLM conflating hardware error with recoverable hang |\n| `fr_rank_correct=no_dumps` | NCCL watchdog did not fire before job ended — adjust `TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC` |\n| `fr_rank_correct=node` | FR isolated exactly one `GPUS_PER_NODE` rank block containing the injected rank, but not the exact rank |\n| `fr_rank_correct=segment` | FR isolated the configured `FR_SEGMENT_SIZE` rank block containing the injected rank, but not the exact node\u002Frank |\n\n---\n\n## Step 6 — Identify and Apply Improvements\n\n### FR analysis\nDeterministic graph algorithm — **do not modify automatically**.\nNote misidentifications and escalate to the team.\n\n### Log analysis (safe to modify)\n\n| Observation | Target location | Suggested fix |\n|---|---|---|\n| Wrong restart for hang | `nvrx_logsage.py` fast-path | Strengthen NCCL timeout → `RESTART IMMEDIATE` mapping |\n| Missing rank in attr text | `nvrx_logsage.py` prompt | Extract rank from NCCL watchdog lines; add regex |\n| Crash misclassified as hang | `nvrx_logsage.py` | Add SIGKILL\u002FSEGFAULT\u002FGPU_ERROR keyword patterns |\n| `ERRORS NOT FOUND` when errors exist | `return_application_errors` config | Loosen error extraction filter |\n| rank-0 not detected | prompt or fast-path | Add explicit rank-0 hang heuristic (other ranks silent) |\n| attr off by many iters | prompt context | Increase weight of iteration-stamped log lines |\n| LLM wrong on GPU_ERROR | prompt | Distinguish `cudaError` → crash from NCCL timeout → hang |\n\nEditable file: `attribution\u002Flog_analyzer\u002Fnvrx_logsage.py`\n\nAfter each patch, re-run the same pool subset that previously failed:\n```bash\nPOOL=\"GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2\" bash scripts\u002Fprepare_node_alloc.sh\n```\n\n---\n\n## Step 7 — Loop\n\nIncrement experiment counter. Suggested sweep order across code-change iterations:\n\n1. **Iteration 1**: full default pool (34 experiments)\n2. **Iteration 2**: targeted re-run of all failing cells from iteration 1\n3. **Iteration 3**: expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps\n4. **Iteration 4**: add SEGFAULT and LOCK_GIL 4-node\u002F8-node coverage\n\nStop condition: all cells pass all four scoring dimensions for two consecutive\ncode-change iterations.\n\n---\n\n## Adapting A SLURM Script For The Feedback Loop\n\nThe feedback loop is not tied to `l4_gb200_reduced.sh`, but your sbatch script must\nmatch a small contract so the loop can submit, analyze, and score each run.\n\nRequired changes for a custom workload script:\n\n1. Accept these exported variables from `prepare_node_alloc.sh`:\n   `FAULT_TYPE`, `FAULT_RANK`, `FAULT_AT_ITER`, `EXPERIMENT_DIR`, `BASE_EXPERIMENTS_DIR`,\n   and `GPUS_PER_NODE`.\n2. Write the main training log to:\n   `${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F${SLURM_JOB_ID}.*.1.main_workload.log`\n   so `watch_and_analyze.sh` can find it.\n3. Write NCCL flight-recorder dumps under `${EXPERIMENT_DIR}\u002Fcheckpoints\u002F`.\n4. Emit a fault-injection marker when the fault is injected.\n   `watch_and_analyze.sh` uses this to decide whether the run reached the injection point.\n5. Preserve the per-experiment directory layout:\n   `logs\u002Fslurm\u002F`, `checkpoints\u002F`, and `tensorboard\u002F`.\n\nThis has only been validated with Megatron-LM because the current run-valid check and\nfault markers depend on Megatron's `debug_fault_injection.py` behavior. If you adapt the\nloop to another framework, update both the sbatch script and `watch_and_analyze.sh`.\n\n## Appendix A: SBATCH_SCRIPT fault parameters\n\nThe example `SBATCH_SCRIPT` reads these env vars from `prepare_node_alloc.sh` via `--export`:\n\n| Variable | Default | Description |\n|---|---|---|\n| `FAULT_AT_ITER` | `5` | Training iteration at which to inject |\n| `FAULT_DELAY` | `15` | Delay in seconds before fault injection after the iteration anchor |\n| `FAULT_RANK` | `1` | Global rank to inject `[0, total_ranks)` |\n| `FAULT_TYPE` | `GPU_SLEEP` | Megatron fault type enum name |\n| `GPUS_PER_NODE` | `4` | GPUs per node (used to compute `TOTAL_TASKS`) |\n| `EXPERIMENT_DIR` | `${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002Fn${SLURM_NNODES}_${FAULT_TYPE}_r${FAULT_RANK}_i${FAULT_AT_ITER}` | Per-experiment output root |\n| `BASE_EXPERIMENTS_DIR` | `${HOME}\u002Fnvrx-attr-experiments` | Shared root (datacache, triton\u002Finductor caches) |\n\nValid `FAULT_TYPE` values:\n`GPU_ERROR`, `GPU_SLEEP`, `WORKLOAD_EXC`, `ASYNC_EXC`, `SIGNAL_EXC`, `OS_ABORT`,\n`LOCK_GIL`, `SEGFAULT`, `SIGINT`, `SIGKILL`, `SIGTERM`, `SIGSTOP`\n\n---\n\n## Appendix B: Single-experiment manual run\n\n```bash\n# Manual runs land under fault_injection\u002Fmanual\u002F by default (no session dir needed)\nEXPERIMENT_DIR=${HOME}\u002Fnvrx-attr-experiments\u002Ffault_injection\u002Fmanual\u002Fn2_GPU_SLEEP_r1_i5\nmkdir -p ${EXPERIMENT_DIR}\u002Flogs\u002Fslurm ${EXPERIMENT_DIR}\u002Fcheckpoints ${EXPERIMENT_DIR}\u002Ftensorboard\n\nsbatch \\\n    --nodes=2 \\\n    --output=${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F%j.launch.out \\\n    --error=${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F%j.launch.err \\\n    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=${EXPERIMENT_DIR} \\\n    scripts\u002Fl4_gb200_reduced.sh\n```\n\nOptional site-specific cleanup:\n\n```bash\nexport CONTAINER_CLEANUP_CMD='\nENROOT_DIR=\"\u002Fvar\u002Flib\u002Fenroot\u002Fdata\u002F$(id -u)\"\nrm -rf \"${ENROOT_DIR:?}\"\u002F* 2>\u002Fdev\u002Fnull || true\necho \"$(hostname): \u002F $(df -h \u002F | tail -1 | awk \"{print \\$3\\\" used, \\\"\\$4\\\" avail\\\"}\")\"\n'\n```\n",{"data":35,"body":41},{"name":4,"description":6,"compatibility":36,"metadata":37},"Requires SLURM cluster access, sbatch, LLM_API_KEY, langchain-openai, logsage, and nvidia-resiliency-ext installed. This workflow has only been validated with Megatron-LM workloads.",{"author":8,"sub-skills":38},[39,40],"log-analysis","fr-analysis",{"type":42,"children":43},"root",[44,53,59,63,70,83,86,92,113,122,127,266,279,331,337,342,367,419,424,801,857,899,904,1474,1487,1583,1588,1613,1702,1705,1711,1729,1742,1751,1756,1765,1776,1779,1785,1790,1860,1865,1937,1942,2057,2060,2066,2079,2337,2342,2433,2474,2477,2483,2495,2507,2516,2521,2695,2698,2704,2709,2721,2727,2922,2933,2938,2977,2980,2986,2991,3034,3039,3042,3048,3060,3065,3188,3207,3213,3240,3448,3542,3545,3551,3775,3780,3841],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"skill-fault-injection-loop",[50],{"type":51,"value":52},"text","Skill: fault-injection-loop",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57],{"type":51,"value":58},"Iterative closed-loop skill that runs a prioritized fault-injection experiment pool\n2 jobs at a time, analyzes every artifact, scores attribution accuracy, aggregates\ngaps across the matrix, and proposes targeted improvements to attribution modules.",{"type":45,"tag":60,"props":61,"children":62},"hr",{},[],{"type":45,"tag":64,"props":65,"children":67},"h2",{"id":66},"overview",[68],{"type":51,"value":69},"Overview",{"type":45,"tag":71,"props":72,"children":76},"pre",{"className":73,"code":75,"language":51},[74],"language-text","┌───────────────────────────────────────────────────────────────────────┐\n│  0. POOL     → build ordered pool of (fault_type, rank, iter, nodes)  │\n│               GPU faults first, then crash, Python-hang, signal       │\n│                                                                        │\n│  repeat until pool exhausted:                                          │\n│  1. SUBMIT   → sbatch 2 jobs from pool head                            │\n│  2. WAIT     → poll until both jobs leave RUNNING\u002FPENDING              │\n│                                                                        │\n│  after all jobs done:                                                  │\n│  3. ANALYZE  → watch_and_analyze.sh: \u002Flog-analysis + \u002Ffr-analysis     │\n│               per completed job, streaming as jobs finish              │\n│  4. SCORE    → compare attribution output vs injected ground truth     │\n│  5. AGGREGATE→ build results table; identify systematic failure modes  │\n│  6. IMPROVE  → patch log_analyzer\u002Fnvrx_logsage.py                     │\n│  7. LOOP     → re-run same pool with updated attribution code          │\n└───────────────────────────────────────────────────────────────────────┘\n",[77],{"type":45,"tag":78,"props":79,"children":81},"code",{"__ignoreMap":80},"",[82],{"type":51,"value":75},{"type":45,"tag":60,"props":84,"children":85},{},[],{"type":45,"tag":64,"props":87,"children":89},{"id":88},"step-0-fault-pool-design",[90],{"type":51,"value":91},"Step 0 — Fault Pool Design",{"type":45,"tag":54,"props":93,"children":94},{},[95,97,103,105,111],{"type":51,"value":96},"The pool is defined as an ordered list of ",{"type":45,"tag":78,"props":98,"children":100},{"className":99},[],[101],{"type":51,"value":102},"FAULT_TYPE:RANK:ITER:NODES",{"type":51,"value":104}," entries\ninside ",{"type":45,"tag":78,"props":106,"children":108},{"className":107},[],[109],{"type":51,"value":110},"scripts\u002Fprepare_node_alloc.sh",{"type":51,"value":112},". Default pool (34 experiments, 17 batches):",{"type":45,"tag":71,"props":114,"children":117},{"className":115,"code":116,"language":51},[74],"# GPU hangs — highest priority; full rank sweep across all node counts\nGPU_SLEEP:1:5:2   GPU_SLEEP:0:5:2      # 2-node: rank-1, rank-0\nGPU_SLEEP:4:5:2   GPU_SLEEP:7:5:2      # 2-node: mid-rank, last-rank\nGPU_SLEEP:1:5:4   GPU_SLEEP:0:5:4      # 4-node: rank-1, rank-0\nGPU_SLEEP:8:5:4   GPU_SLEEP:15:5:4     # 4-node: mid, last\nGPU_SLEEP:1:5:8   GPU_SLEEP:0:5:8      # 8-node: rank-1, rank-0\nGPU_SLEEP:16:5:8  GPU_SLEEP:31:5:8     # 8-node: mid, last\n\n# GPU errors — high priority; rank-0 and rank-1 across all node counts\nGPU_ERROR:1:5:2   GPU_ERROR:0:5:2\nGPU_ERROR:1:5:4   GPU_ERROR:0:5:4\nGPU_ERROR:1:5:8   GPU_ERROR:0:5:8\n\n# Crash faults\nSIGKILL:1:5:2     SIGKILL:0:5:2\nSIGKILL:1:5:4     SIGKILL:1:5:8\nSEGFAULT:1:5:2    SEGFAULT:0:5:2\nSEGFAULT:1:5:4    OS_ABORT:1:5:2\n\n# Python-level hangs\nLOCK_GIL:1:5:2    LOCK_GIL:0:5:2\nWORKLOAD_EXC:1:5:2  ASYNC_EXC:1:5:2\n\n# Signals\nSIGTERM:1:5:2     SIGINT:1:5:2\nSIGSTOP:1:5:2     SIGNAL_EXC:1:5:2\n",[118],{"type":45,"tag":78,"props":119,"children":120},{"__ignoreMap":80},[121],{"type":51,"value":116},{"type":45,"tag":54,"props":123,"children":124},{},[125],{"type":51,"value":126},"Rank coverage per node count (4 GPUs\u002Fnode):",{"type":45,"tag":128,"props":129,"children":130},"table",{},[131,170],{"type":45,"tag":132,"props":133,"children":134},"thead",{},[135],{"type":45,"tag":136,"props":137,"children":138},"tr",{},[139,145,150,155,160,165],{"type":45,"tag":140,"props":141,"children":142},"th",{},[143],{"type":51,"value":144},"Nodes",{"type":45,"tag":140,"props":146,"children":147},{},[148],{"type":51,"value":149},"Total ranks",{"type":45,"tag":140,"props":151,"children":152},{},[153],{"type":51,"value":154},"rank-0",{"type":45,"tag":140,"props":156,"children":157},{},[158],{"type":51,"value":159},"rank-1",{"type":45,"tag":140,"props":161,"children":162},{},[163],{"type":51,"value":164},"mid",{"type":45,"tag":140,"props":166,"children":167},{},[168],{"type":51,"value":169},"last",{"type":45,"tag":171,"props":172,"children":173},"tbody",{},[174,208,237],{"type":45,"tag":136,"props":175,"children":176},{},[177,183,188,193,198,203],{"type":45,"tag":178,"props":179,"children":180},"td",{},[181],{"type":51,"value":182},"2",{"type":45,"tag":178,"props":184,"children":185},{},[186],{"type":51,"value":187},"8",{"type":45,"tag":178,"props":189,"children":190},{},[191],{"type":51,"value":192},"0",{"type":45,"tag":178,"props":194,"children":195},{},[196],{"type":51,"value":197},"1",{"type":45,"tag":178,"props":199,"children":200},{},[201],{"type":51,"value":202},"4",{"type":45,"tag":178,"props":204,"children":205},{},[206],{"type":51,"value":207},"7",{"type":45,"tag":136,"props":209,"children":210},{},[211,215,220,224,228,232],{"type":45,"tag":178,"props":212,"children":213},{},[214],{"type":51,"value":202},{"type":45,"tag":178,"props":216,"children":217},{},[218],{"type":51,"value":219},"16",{"type":45,"tag":178,"props":221,"children":222},{},[223],{"type":51,"value":192},{"type":45,"tag":178,"props":225,"children":226},{},[227],{"type":51,"value":197},{"type":45,"tag":178,"props":229,"children":230},{},[231],{"type":51,"value":187},{"type":45,"tag":178,"props":233,"children":234},{},[235],{"type":51,"value":236},"15",{"type":45,"tag":136,"props":238,"children":239},{},[240,244,249,253,257,261],{"type":45,"tag":178,"props":241,"children":242},{},[243],{"type":51,"value":187},{"type":45,"tag":178,"props":245,"children":246},{},[247],{"type":51,"value":248},"32",{"type":45,"tag":178,"props":250,"children":251},{},[252],{"type":51,"value":192},{"type":45,"tag":178,"props":254,"children":255},{},[256],{"type":51,"value":197},{"type":45,"tag":178,"props":258,"children":259},{},[260],{"type":51,"value":219},{"type":45,"tag":178,"props":262,"children":263},{},[264],{"type":51,"value":265},"31",{"type":45,"tag":54,"props":267,"children":268},{},[269,271,277],{"type":51,"value":270},"To run a custom subset, override ",{"type":45,"tag":78,"props":272,"children":274},{"className":273},[],[275],{"type":51,"value":276},"POOL",{"type":51,"value":278}," before calling the script:",{"type":45,"tag":71,"props":280,"children":284},{"className":281,"code":282,"language":283,"meta":80,"style":80},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","POOL=\"GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2\" bash scripts\u002Fprepare_node_alloc.sh\n","bash",[285],{"type":45,"tag":78,"props":286,"children":287},{"__ignoreMap":80},[288],{"type":45,"tag":289,"props":290,"children":293},"span",{"class":291,"line":292},"line",1,[294,299,305,310,316,320,326],{"type":45,"tag":289,"props":295,"children":297},{"style":296},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[298],{"type":51,"value":276},{"type":45,"tag":289,"props":300,"children":302},{"style":301},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[303],{"type":51,"value":304},"=",{"type":45,"tag":289,"props":306,"children":307},{"style":301},[308],{"type":51,"value":309},"\"",{"type":45,"tag":289,"props":311,"children":313},{"style":312},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[314],{"type":51,"value":315},"GPU_SLEEP:0:5:2 GPU_SLEEP:1:5:2",{"type":45,"tag":289,"props":317,"children":318},{"style":301},[319],{"type":51,"value":309},{"type":45,"tag":289,"props":321,"children":323},{"style":322},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[324],{"type":51,"value":325}," bash",{"type":45,"tag":289,"props":327,"children":328},{"style":312},[329],{"type":51,"value":330}," scripts\u002Fprepare_node_alloc.sh\n",{"type":45,"tag":64,"props":332,"children":334},{"id":333},"local-user-config",[335],{"type":51,"value":336},"Local User Config",{"type":45,"tag":54,"props":338,"children":339},{},[340],{"type":51,"value":341},"Start from the tracked template:",{"type":45,"tag":71,"props":343,"children":345},{"className":281,"code":344,"language":283,"meta":80,"style":80},"cp scripts\u002Fuser.env.example scripts\u002Fuser.env\n",[346],{"type":45,"tag":78,"props":347,"children":348},{"__ignoreMap":80},[349],{"type":45,"tag":289,"props":350,"children":351},{"class":291,"line":292},[352,357,362],{"type":45,"tag":289,"props":353,"children":354},{"style":322},[355],{"type":51,"value":356},"cp",{"type":45,"tag":289,"props":358,"children":359},{"style":312},[360],{"type":51,"value":361}," scripts\u002Fuser.env.example",{"type":45,"tag":289,"props":363,"children":364},{"style":312},[365],{"type":51,"value":366}," scripts\u002Fuser.env\n",{"type":45,"tag":54,"props":368,"children":369},{},[370,372,378,380,386,388,394,395,401,403,409,411,417],{"type":51,"value":371},"Then edit ",{"type":45,"tag":78,"props":373,"children":375},{"className":374},[],[376],{"type":51,"value":377},"scripts\u002Fuser.env",{"type":51,"value":379}," with cluster-specific settings. This file is\nsourced by ",{"type":45,"tag":78,"props":381,"children":383},{"className":382},[],[384],{"type":51,"value":385},"run_session.sh",{"type":51,"value":387},", ",{"type":45,"tag":78,"props":389,"children":391},{"className":390},[],[392],{"type":51,"value":393},"prepare_node_alloc.sh",{"type":51,"value":387},{"type":45,"tag":78,"props":396,"children":398},{"className":397},[],[399],{"type":51,"value":400},"watch_and_analyze.sh",{"type":51,"value":402},",\n",{"type":45,"tag":78,"props":404,"children":406},{"className":405},[],[407],{"type":51,"value":408},"l4_gb200_reduced.sh",{"type":51,"value":410},", and ",{"type":45,"tag":78,"props":412,"children":414},{"className":413},[],[415],{"type":51,"value":416},"n3_super_gb200_fi.sh",{"type":51,"value":418},". It is required for this skill\nto run and is intended to stay local and untracked.",{"type":45,"tag":54,"props":420,"children":421},{},[422],{"type":51,"value":423},"Recommended contents:",{"type":45,"tag":71,"props":425,"children":427},{"className":281,"code":426,"language":283,"meta":80,"style":80},"PARTITION=gb-nvl-134-135\nBASE_EXPERIMENTS_DIR=\"${HOME}\u002Fnvrx-attr-experiments\"\nMEGATRON_REPO_HOST_PATH=\"${HOME}\u002Fmegatron-lm-main\"\nSHARED_TMP_BASE_DIR=\"${HOME}\u002Ftmp\"\nWORKSPACE_HOST_PATH=\"${HOME}\u002Ftmp\"\nCONTAINER_IMAGE=\"nvcr.io\u002Fnvidia\u002Fnemo:26.04\"\nLLM_API_KEY_FILE=\"${HOME}\u002F.llm_api_key\"\nJUDGE_API_KEY_FILE=\"${HOME}\u002F.llm_api_key\"\nNVRX_LLM_MODEL=\"nvidia\u002Fnemotron-3-super-120b-a12b\"\nNVRX_LLM_BASE_URL=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\"\nJUDGE_MODEL=\"qwen\u002Fqwen3.5-397b-a17b\"\nJUDGE_BASE_URL=\"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1\"\nFR_SEGMENT_SIZE=32\n",[428],{"type":45,"tag":78,"props":429,"children":430},{"__ignoreMap":80},[431,448,486,520,554,587,613,647,680,706,732,758,783],{"type":45,"tag":289,"props":432,"children":433},{"class":291,"line":292},[434,439,443],{"type":45,"tag":289,"props":435,"children":436},{"style":296},[437],{"type":51,"value":438},"PARTITION",{"type":45,"tag":289,"props":440,"children":441},{"style":301},[442],{"type":51,"value":304},{"type":45,"tag":289,"props":444,"children":445},{"style":312},[446],{"type":51,"value":447},"gb-nvl-134-135\n",{"type":45,"tag":289,"props":449,"children":451},{"class":291,"line":450},2,[452,457,461,466,471,476,481],{"type":45,"tag":289,"props":453,"children":454},{"style":296},[455],{"type":51,"value":456},"BASE_EXPERIMENTS_DIR",{"type":45,"tag":289,"props":458,"children":459},{"style":301},[460],{"type":51,"value":304},{"type":45,"tag":289,"props":462,"children":463},{"style":301},[464],{"type":51,"value":465},"\"${",{"type":45,"tag":289,"props":467,"children":468},{"style":296},[469],{"type":51,"value":470},"HOME",{"type":45,"tag":289,"props":472,"children":473},{"style":301},[474],{"type":51,"value":475},"}",{"type":45,"tag":289,"props":477,"children":478},{"style":312},[479],{"type":51,"value":480},"\u002Fnvrx-attr-experiments",{"type":45,"tag":289,"props":482,"children":483},{"style":301},[484],{"type":51,"value":485},"\"\n",{"type":45,"tag":289,"props":487,"children":489},{"class":291,"line":488},3,[490,495,499,503,507,511,516],{"type":45,"tag":289,"props":491,"children":492},{"style":296},[493],{"type":51,"value":494},"MEGATRON_REPO_HOST_PATH",{"type":45,"tag":289,"props":496,"children":497},{"style":301},[498],{"type":51,"value":304},{"type":45,"tag":289,"props":500,"children":501},{"style":301},[502],{"type":51,"value":465},{"type":45,"tag":289,"props":504,"children":505},{"style":296},[506],{"type":51,"value":470},{"type":45,"tag":289,"props":508,"children":509},{"style":301},[510],{"type":51,"value":475},{"type":45,"tag":289,"props":512,"children":513},{"style":312},[514],{"type":51,"value":515},"\u002Fmegatron-lm-main",{"type":45,"tag":289,"props":517,"children":518},{"style":301},[519],{"type":51,"value":485},{"type":45,"tag":289,"props":521,"children":523},{"class":291,"line":522},4,[524,529,533,537,541,545,550],{"type":45,"tag":289,"props":525,"children":526},{"style":296},[527],{"type":51,"value":528},"SHARED_TMP_BASE_DIR",{"type":45,"tag":289,"props":530,"children":531},{"style":301},[532],{"type":51,"value":304},{"type":45,"tag":289,"props":534,"children":535},{"style":301},[536],{"type":51,"value":465},{"type":45,"tag":289,"props":538,"children":539},{"style":296},[540],{"type":51,"value":470},{"type":45,"tag":289,"props":542,"children":543},{"style":301},[544],{"type":51,"value":475},{"type":45,"tag":289,"props":546,"children":547},{"style":312},[548],{"type":51,"value":549},"\u002Ftmp",{"type":45,"tag":289,"props":551,"children":552},{"style":301},[553],{"type":51,"value":485},{"type":45,"tag":289,"props":555,"children":557},{"class":291,"line":556},5,[558,563,567,571,575,579,583],{"type":45,"tag":289,"props":559,"children":560},{"style":296},[561],{"type":51,"value":562},"WORKSPACE_HOST_PATH",{"type":45,"tag":289,"props":564,"children":565},{"style":301},[566],{"type":51,"value":304},{"type":45,"tag":289,"props":568,"children":569},{"style":301},[570],{"type":51,"value":465},{"type":45,"tag":289,"props":572,"children":573},{"style":296},[574],{"type":51,"value":470},{"type":45,"tag":289,"props":576,"children":577},{"style":301},[578],{"type":51,"value":475},{"type":45,"tag":289,"props":580,"children":581},{"style":312},[582],{"type":51,"value":549},{"type":45,"tag":289,"props":584,"children":585},{"style":301},[586],{"type":51,"value":485},{"type":45,"tag":289,"props":588,"children":590},{"class":291,"line":589},6,[591,596,600,604,609],{"type":45,"tag":289,"props":592,"children":593},{"style":296},[594],{"type":51,"value":595},"CONTAINER_IMAGE",{"type":45,"tag":289,"props":597,"children":598},{"style":301},[599],{"type":51,"value":304},{"type":45,"tag":289,"props":601,"children":602},{"style":301},[603],{"type":51,"value":309},{"type":45,"tag":289,"props":605,"children":606},{"style":312},[607],{"type":51,"value":608},"nvcr.io\u002Fnvidia\u002Fnemo:26.04",{"type":45,"tag":289,"props":610,"children":611},{"style":301},[612],{"type":51,"value":485},{"type":45,"tag":289,"props":614,"children":616},{"class":291,"line":615},7,[617,622,626,630,634,638,643],{"type":45,"tag":289,"props":618,"children":619},{"style":296},[620],{"type":51,"value":621},"LLM_API_KEY_FILE",{"type":45,"tag":289,"props":623,"children":624},{"style":301},[625],{"type":51,"value":304},{"type":45,"tag":289,"props":627,"children":628},{"style":301},[629],{"type":51,"value":465},{"type":45,"tag":289,"props":631,"children":632},{"style":296},[633],{"type":51,"value":470},{"type":45,"tag":289,"props":635,"children":636},{"style":301},[637],{"type":51,"value":475},{"type":45,"tag":289,"props":639,"children":640},{"style":312},[641],{"type":51,"value":642},"\u002F.llm_api_key",{"type":45,"tag":289,"props":644,"children":645},{"style":301},[646],{"type":51,"value":485},{"type":45,"tag":289,"props":648,"children":650},{"class":291,"line":649},8,[651,656,660,664,668,672,676],{"type":45,"tag":289,"props":652,"children":653},{"style":296},[654],{"type":51,"value":655},"JUDGE_API_KEY_FILE",{"type":45,"tag":289,"props":657,"children":658},{"style":301},[659],{"type":51,"value":304},{"type":45,"tag":289,"props":661,"children":662},{"style":301},[663],{"type":51,"value":465},{"type":45,"tag":289,"props":665,"children":666},{"style":296},[667],{"type":51,"value":470},{"type":45,"tag":289,"props":669,"children":670},{"style":301},[671],{"type":51,"value":475},{"type":45,"tag":289,"props":673,"children":674},{"style":312},[675],{"type":51,"value":642},{"type":45,"tag":289,"props":677,"children":678},{"style":301},[679],{"type":51,"value":485},{"type":45,"tag":289,"props":681,"children":683},{"class":291,"line":682},9,[684,689,693,697,702],{"type":45,"tag":289,"props":685,"children":686},{"style":296},[687],{"type":51,"value":688},"NVRX_LLM_MODEL",{"type":45,"tag":289,"props":690,"children":691},{"style":301},[692],{"type":51,"value":304},{"type":45,"tag":289,"props":694,"children":695},{"style":301},[696],{"type":51,"value":309},{"type":45,"tag":289,"props":698,"children":699},{"style":312},[700],{"type":51,"value":701},"nvidia\u002Fnemotron-3-super-120b-a12b",{"type":45,"tag":289,"props":703,"children":704},{"style":301},[705],{"type":51,"value":485},{"type":45,"tag":289,"props":707,"children":709},{"class":291,"line":708},10,[710,715,719,723,728],{"type":45,"tag":289,"props":711,"children":712},{"style":296},[713],{"type":51,"value":714},"NVRX_LLM_BASE_URL",{"type":45,"tag":289,"props":716,"children":717},{"style":301},[718],{"type":51,"value":304},{"type":45,"tag":289,"props":720,"children":721},{"style":301},[722],{"type":51,"value":309},{"type":45,"tag":289,"props":724,"children":725},{"style":312},[726],{"type":51,"value":727},"https:\u002F\u002Fintegrate.api.nvidia.com\u002Fv1",{"type":45,"tag":289,"props":729,"children":730},{"style":301},[731],{"type":51,"value":485},{"type":45,"tag":289,"props":733,"children":735},{"class":291,"line":734},11,[736,741,745,749,754],{"type":45,"tag":289,"props":737,"children":738},{"style":296},[739],{"type":51,"value":740},"JUDGE_MODEL",{"type":45,"tag":289,"props":742,"children":743},{"style":301},[744],{"type":51,"value":304},{"type":45,"tag":289,"props":746,"children":747},{"style":301},[748],{"type":51,"value":309},{"type":45,"tag":289,"props":750,"children":751},{"style":312},[752],{"type":51,"value":753},"qwen\u002Fqwen3.5-397b-a17b",{"type":45,"tag":289,"props":755,"children":756},{"style":301},[757],{"type":51,"value":485},{"type":45,"tag":289,"props":759,"children":761},{"class":291,"line":760},12,[762,767,771,775,779],{"type":45,"tag":289,"props":763,"children":764},{"style":296},[765],{"type":51,"value":766},"JUDGE_BASE_URL",{"type":45,"tag":289,"props":768,"children":769},{"style":301},[770],{"type":51,"value":304},{"type":45,"tag":289,"props":772,"children":773},{"style":301},[774],{"type":51,"value":309},{"type":45,"tag":289,"props":776,"children":777},{"style":312},[778],{"type":51,"value":727},{"type":45,"tag":289,"props":780,"children":781},{"style":301},[782],{"type":51,"value":485},{"type":45,"tag":289,"props":784,"children":786},{"class":291,"line":785},13,[787,792,796],{"type":45,"tag":289,"props":788,"children":789},{"style":296},[790],{"type":51,"value":791},"FR_SEGMENT_SIZE",{"type":45,"tag":289,"props":793,"children":794},{"style":301},[795],{"type":51,"value":304},{"type":45,"tag":289,"props":797,"children":798},{"style":312},[799],{"type":51,"value":800},"32\n",{"type":45,"tag":54,"props":802,"children":803},{},[804,806,812,814,819,820,826,827,833,834,840,841,847,849,855],{"type":51,"value":805},"Use ",{"type":45,"tag":78,"props":807,"children":809},{"className":808},[],[810],{"type":51,"value":811},"user.env",{"type":51,"value":813}," for stable site defaults such as partition, container image, and\nhost paths, plus local LLM credentials and endpoint settings for log-analysis\nand the judge. Use per-run environment overrides for experiment-specific\ncontrols such as ",{"type":45,"tag":78,"props":815,"children":817},{"className":816},[],[818],{"type":51,"value":276},{"type":51,"value":387},{"type":45,"tag":78,"props":821,"children":823},{"className":822},[],[824],{"type":51,"value":825},"WORKLOAD",{"type":51,"value":387},{"type":45,"tag":78,"props":828,"children":830},{"className":829},[],[831],{"type":51,"value":832},"BATCH_SIZE",{"type":51,"value":387},{"type":45,"tag":78,"props":835,"children":837},{"className":836},[],[838],{"type":51,"value":839},"FAULT_TYPE",{"type":51,"value":402},{"type":45,"tag":78,"props":842,"children":844},{"className":843},[],[845],{"type":51,"value":846},"FAULT_AT_ITER",{"type":51,"value":848},", or ",{"type":45,"tag":78,"props":850,"children":852},{"className":851},[],[853],{"type":51,"value":854},"FAULT_DELAY",{"type":51,"value":856},".",{"type":45,"tag":54,"props":858,"children":859},{},[860,862,867,869,875,877,883,884,890,891,897],{"type":51,"value":861},"If you use local Triton\u002FInductor cache staging, set the cache variables in\n",{"type":45,"tag":78,"props":863,"children":865},{"className":864},[],[866],{"type":51,"value":377},{"type":51,"value":868},". See ",{"type":45,"tag":78,"props":870,"children":872},{"className":871},[],[873],{"type":51,"value":874},"scripts\u002Fuser.env.example",{"type":51,"value":876}," for the supported\n",{"type":45,"tag":78,"props":878,"children":880},{"className":879},[],[881],{"type":51,"value":882},"ENABLE_NFS_CACHE_STAGING",{"type":51,"value":387},{"type":45,"tag":78,"props":885,"children":887},{"className":886},[],[888],{"type":51,"value":889},"NFS_TRITON_CACHE",{"type":51,"value":410},{"type":45,"tag":78,"props":892,"children":894},{"className":893},[],[895],{"type":51,"value":896},"NFS_INDUCTOR_CACHE",{"type":51,"value":898},"\nentries and workload-specific path examples.",{"type":45,"tag":54,"props":900,"children":901},{},[902],{"type":51,"value":903},"Environment variables:",{"type":45,"tag":128,"props":905,"children":906},{},[907,928],{"type":45,"tag":132,"props":908,"children":909},{},[910],{"type":45,"tag":136,"props":911,"children":912},{},[913,918,923],{"type":45,"tag":140,"props":914,"children":915},{},[916],{"type":51,"value":917},"Variable",{"type":45,"tag":140,"props":919,"children":920},{},[921],{"type":51,"value":922},"Default",{"type":45,"tag":140,"props":924,"children":925},{},[926],{"type":51,"value":927},"Description",{"type":45,"tag":171,"props":929,"children":930},{},[931,964,996,1025,1050,1076,1100,1126,1151,1176,1201,1233,1257,1281,1304,1328,1352,1376,1400,1424,1450],{"type":45,"tag":136,"props":932,"children":933},{},[934,942,951],{"type":45,"tag":178,"props":935,"children":936},{},[937],{"type":45,"tag":78,"props":938,"children":940},{"className":939},[],[941],{"type":51,"value":825},{"type":45,"tag":178,"props":943,"children":944},{},[945],{"type":45,"tag":78,"props":946,"children":948},{"className":947},[],[949],{"type":51,"value":950},"llama4_scout",{"type":45,"tag":178,"props":952,"children":953},{},[954,956,962],{"type":51,"value":955},"Select a registered workload by name (see ",{"type":45,"tag":78,"props":957,"children":959},{"className":958},[],[960],{"type":51,"value":961},"scripts\u002Fworkloads.conf",{"type":51,"value":963},")",{"type":45,"tag":136,"props":965,"children":966},{},[967,976,991],{"type":45,"tag":178,"props":968,"children":969},{},[970],{"type":45,"tag":78,"props":971,"children":973},{"className":972},[],[974],{"type":51,"value":975},"ACCOUNT",{"type":45,"tag":178,"props":977,"children":978},{},[979],{"type":45,"tag":980,"props":981,"children":982},"em",{},[983,985,990],{"type":51,"value":984},"(cluster default or ",{"type":45,"tag":78,"props":986,"children":988},{"className":987},[],[989],{"type":51,"value":377},{"type":51,"value":963},{"type":45,"tag":178,"props":992,"children":993},{},[994],{"type":51,"value":995},"SLURM account",{"type":45,"tag":136,"props":997,"children":998},{},[999,1007,1020],{"type":45,"tag":178,"props":1000,"children":1001},{},[1002],{"type":45,"tag":78,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":51,"value":438},{"type":45,"tag":178,"props":1008,"children":1009},{},[1010],{"type":45,"tag":980,"props":1011,"children":1012},{},[1013,1014,1019],{"type":51,"value":984},{"type":45,"tag":78,"props":1015,"children":1017},{"className":1016},[],[1018],{"type":51,"value":377},{"type":51,"value":963},{"type":45,"tag":178,"props":1021,"children":1022},{},[1023],{"type":51,"value":1024},"SLURM partition",{"type":45,"tag":136,"props":1026,"children":1027},{},[1028,1037,1045],{"type":45,"tag":178,"props":1029,"children":1030},{},[1031],{"type":45,"tag":78,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":51,"value":1036},"GPUS_PER_NODE",{"type":45,"tag":178,"props":1038,"children":1039},{},[1040],{"type":45,"tag":78,"props":1041,"children":1043},{"className":1042},[],[1044],{"type":51,"value":202},{"type":45,"tag":178,"props":1046,"children":1047},{},[1048],{"type":51,"value":1049},"GPUs per node",{"type":45,"tag":136,"props":1051,"children":1052},{},[1053,1062,1071],{"type":45,"tag":178,"props":1054,"children":1055},{},[1056],{"type":45,"tag":78,"props":1057,"children":1059},{"className":1058},[],[1060],{"type":51,"value":1061},"TIME",{"type":45,"tag":178,"props":1063,"children":1064},{},[1065],{"type":45,"tag":78,"props":1066,"children":1068},{"className":1067},[],[1069],{"type":51,"value":1070},"00:30:00",{"type":45,"tag":178,"props":1072,"children":1073},{},[1074],{"type":51,"value":1075},"Per-job wall-clock limit",{"type":45,"tag":136,"props":1077,"children":1078},{},[1079,1087,1095],{"type":45,"tag":178,"props":1080,"children":1081},{},[1082],{"type":45,"tag":78,"props":1083,"children":1085},{"className":1084},[],[1086],{"type":51,"value":832},{"type":45,"tag":178,"props":1088,"children":1089},{},[1090],{"type":45,"tag":78,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":51,"value":182},{"type":45,"tag":178,"props":1096,"children":1097},{},[1098],{"type":51,"value":1099},"Jobs submitted per round",{"type":45,"tag":136,"props":1101,"children":1102},{},[1103,1112,1121],{"type":45,"tag":178,"props":1104,"children":1105},{},[1106],{"type":45,"tag":78,"props":1107,"children":1109},{"className":1108},[],[1110],{"type":51,"value":1111},"POLL_INTERVAL",{"type":45,"tag":178,"props":1113,"children":1114},{},[1115],{"type":45,"tag":78,"props":1116,"children":1118},{"className":1117},[],[1119],{"type":51,"value":1120},"30",{"type":45,"tag":178,"props":1122,"children":1123},{},[1124],{"type":51,"value":1125},"Seconds between queue polls",{"type":45,"tag":136,"props":1127,"children":1128},{},[1129,1137,1146],{"type":45,"tag":178,"props":1130,"children":1131},{},[1132],{"type":45,"tag":78,"props":1133,"children":1135},{"className":1134},[],[1136],{"type":51,"value":456},{"type":45,"tag":178,"props":1138,"children":1139},{},[1140],{"type":45,"tag":78,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":51,"value":1145},"${HOME}\u002Fnvrx-attr-experiments",{"type":45,"tag":178,"props":1147,"children":1148},{},[1149],{"type":51,"value":1150},"Root for all output",{"type":45,"tag":136,"props":1152,"children":1153},{},[1154,1162,1171],{"type":45,"tag":178,"props":1155,"children":1156},{},[1157],{"type":45,"tag":78,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":51,"value":494},{"type":45,"tag":178,"props":1163,"children":1164},{},[1165],{"type":45,"tag":78,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":51,"value":1170},"${HOME}\u002Fmegatron-lm-main",{"type":45,"tag":178,"props":1172,"children":1173},{},[1174],{"type":51,"value":1175},"Host path to the Megatron checkout mounted into the container",{"type":45,"tag":136,"props":1177,"children":1178},{},[1179,1187,1196],{"type":45,"tag":178,"props":1180,"children":1181},{},[1182],{"type":45,"tag":78,"props":1183,"children":1185},{"className":1184},[],[1186],{"type":51,"value":528},{"type":45,"tag":178,"props":1188,"children":1189},{},[1190],{"type":45,"tag":78,"props":1191,"children":1193},{"className":1192},[],[1194],{"type":51,"value":1195},"${HOME}\u002Ftmp",{"type":45,"tag":178,"props":1197,"children":1198},{},[1199],{"type":51,"value":1200},"Shared filesystem path used for cross-step coordination",{"type":45,"tag":136,"props":1202,"children":1203},{},[1204,1212,1220],{"type":45,"tag":178,"props":1205,"children":1206},{},[1207],{"type":45,"tag":78,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":51,"value":562},{"type":45,"tag":178,"props":1213,"children":1214},{},[1215],{"type":45,"tag":78,"props":1216,"children":1218},{"className":1217},[],[1219],{"type":51,"value":1195},{"type":45,"tag":178,"props":1221,"children":1222},{},[1223,1225,1231],{"type":51,"value":1224},"Host path mounted at ",{"type":45,"tag":78,"props":1226,"children":1228},{"className":1227},[],[1229],{"type":51,"value":1230},"\u002Fworkspace",{"type":51,"value":1232}," inside the container",{"type":45,"tag":136,"props":1234,"children":1235},{},[1236,1244,1252],{"type":45,"tag":178,"props":1237,"children":1238},{},[1239],{"type":45,"tag":78,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":51,"value":595},{"type":45,"tag":178,"props":1245,"children":1246},{},[1247],{"type":45,"tag":78,"props":1248,"children":1250},{"className":1249},[],[1251],{"type":51,"value":608},{"type":45,"tag":178,"props":1253,"children":1254},{},[1255],{"type":51,"value":1256},"Container image used by the workload script",{"type":45,"tag":136,"props":1258,"children":1259},{},[1260,1268,1276],{"type":45,"tag":178,"props":1261,"children":1262},{},[1263],{"type":45,"tag":78,"props":1264,"children":1266},{"className":1265},[],[1267],{"type":51,"value":621},{"type":45,"tag":178,"props":1269,"children":1270},{},[1271],{"type":45,"tag":980,"props":1272,"children":1273},{},[1274],{"type":51,"value":1275},"unset",{"type":45,"tag":178,"props":1277,"children":1278},{},[1279],{"type":51,"value":1280},"File containing the log-analysis API key",{"type":45,"tag":136,"props":1282,"children":1283},{},[1284,1292,1299],{"type":45,"tag":178,"props":1285,"children":1286},{},[1287],{"type":45,"tag":78,"props":1288,"children":1290},{"className":1289},[],[1291],{"type":51,"value":655},{"type":45,"tag":178,"props":1293,"children":1294},{},[1295],{"type":45,"tag":980,"props":1296,"children":1297},{},[1298],{"type":51,"value":1275},{"type":45,"tag":178,"props":1300,"children":1301},{},[1302],{"type":51,"value":1303},"File containing the judge API key",{"type":45,"tag":136,"props":1305,"children":1306},{},[1307,1315,1323],{"type":45,"tag":178,"props":1308,"children":1309},{},[1310],{"type":45,"tag":78,"props":1311,"children":1313},{"className":1312},[],[1314],{"type":51,"value":688},{"type":45,"tag":178,"props":1316,"children":1317},{},[1318],{"type":45,"tag":78,"props":1319,"children":1321},{"className":1320},[],[1322],{"type":51,"value":701},{"type":45,"tag":178,"props":1324,"children":1325},{},[1326],{"type":51,"value":1327},"Model for log-analysis",{"type":45,"tag":136,"props":1329,"children":1330},{},[1331,1339,1347],{"type":45,"tag":178,"props":1332,"children":1333},{},[1334],{"type":45,"tag":78,"props":1335,"children":1337},{"className":1336},[],[1338],{"type":51,"value":714},{"type":45,"tag":178,"props":1340,"children":1341},{},[1342],{"type":45,"tag":78,"props":1343,"children":1345},{"className":1344},[],[1346],{"type":51,"value":727},{"type":45,"tag":178,"props":1348,"children":1349},{},[1350],{"type":51,"value":1351},"Base URL for log-analysis",{"type":45,"tag":136,"props":1353,"children":1354},{},[1355,1363,1371],{"type":45,"tag":178,"props":1356,"children":1357},{},[1358],{"type":45,"tag":78,"props":1359,"children":1361},{"className":1360},[],[1362],{"type":51,"value":740},{"type":45,"tag":178,"props":1364,"children":1365},{},[1366],{"type":45,"tag":78,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":51,"value":753},{"type":45,"tag":178,"props":1372,"children":1373},{},[1374],{"type":51,"value":1375},"Model for judge scoring",{"type":45,"tag":136,"props":1377,"children":1378},{},[1379,1387,1395],{"type":45,"tag":178,"props":1380,"children":1381},{},[1382],{"type":45,"tag":78,"props":1383,"children":1385},{"className":1384},[],[1386],{"type":51,"value":766},{"type":45,"tag":178,"props":1388,"children":1389},{},[1390],{"type":45,"tag":78,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":51,"value":727},{"type":45,"tag":178,"props":1396,"children":1397},{},[1398],{"type":51,"value":1399},"Base URL for judge scoring",{"type":45,"tag":136,"props":1401,"children":1402},{},[1403,1411,1419],{"type":45,"tag":178,"props":1404,"children":1405},{},[1406],{"type":45,"tag":78,"props":1407,"children":1409},{"className":1408},[],[1410],{"type":51,"value":791},{"type":45,"tag":178,"props":1412,"children":1413},{},[1414],{"type":45,"tag":78,"props":1415,"children":1417},{"className":1416},[],[1418],{"type":51,"value":248},{"type":45,"tag":178,"props":1420,"children":1421},{},[1422],{"type":51,"value":1423},"Ranks per segment for coarse FR scoring",{"type":45,"tag":136,"props":1425,"children":1426},{},[1427,1436,1445],{"type":45,"tag":178,"props":1428,"children":1429},{},[1430],{"type":45,"tag":78,"props":1431,"children":1433},{"className":1432},[],[1434],{"type":51,"value":1435},"SBATCH_SCRIPT",{"type":45,"tag":178,"props":1437,"children":1438},{},[1439],{"type":45,"tag":78,"props":1440,"children":1442},{"className":1441},[],[1443],{"type":51,"value":1444},"scripts\u002Fl4_gb200_reduced.sh",{"type":45,"tag":178,"props":1446,"children":1447},{},[1448],{"type":51,"value":1449},"Job script to submit",{"type":45,"tag":136,"props":1451,"children":1452},{},[1453,1461,1469],{"type":45,"tag":178,"props":1454,"children":1455},{},[1456],{"type":45,"tag":78,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":51,"value":276},{"type":45,"tag":178,"props":1462,"children":1463},{},[1464],{"type":45,"tag":980,"props":1465,"children":1466},{},[1467],{"type":51,"value":1468},"(default pool above)",{"type":45,"tag":178,"props":1470,"children":1471},{},[1472],{"type":51,"value":1473},"Space-separated experiment triplets",{"type":45,"tag":1475,"props":1476,"children":1478},"h3",{"id":1477},"registered-workloads-scriptsworkloadsconf",[1479,1481,1486],{"type":51,"value":1480},"Registered workloads (",{"type":45,"tag":78,"props":1482,"children":1484},{"className":1483},[],[1485],{"type":51,"value":961},{"type":51,"value":963},{"type":45,"tag":128,"props":1488,"children":1489},{},[1490,1515],{"type":45,"tag":132,"props":1491,"children":1492},{},[1493],{"type":45,"tag":136,"props":1494,"children":1495},{},[1496,1501,1506,1511],{"type":45,"tag":140,"props":1497,"children":1498},{},[1499],{"type":51,"value":1500},"Name",{"type":45,"tag":140,"props":1502,"children":1503},{},[1504],{"type":51,"value":1505},"Script",{"type":45,"tag":140,"props":1507,"children":1508},{},[1509],{"type":51,"value":1510},"Base dir",{"type":45,"tag":140,"props":1512,"children":1513},{},[1514],{"type":51,"value":927},{"type":45,"tag":171,"props":1516,"children":1517},{},[1518,1550],{"type":45,"tag":136,"props":1519,"children":1520},{},[1521,1529,1537,1545],{"type":45,"tag":178,"props":1522,"children":1523},{},[1524],{"type":45,"tag":78,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":51,"value":950},{"type":45,"tag":178,"props":1530,"children":1531},{},[1532],{"type":45,"tag":78,"props":1533,"children":1535},{"className":1534},[],[1536],{"type":51,"value":408},{"type":45,"tag":178,"props":1538,"children":1539},{},[1540],{"type":45,"tag":78,"props":1541,"children":1543},{"className":1542},[],[1544],{"type":51,"value":1145},{"type":45,"tag":178,"props":1546,"children":1547},{},[1548],{"type":51,"value":1549},"Llama4-Scout (reduced layers) on GB200; minimum supported size is 2 nodes",{"type":45,"tag":136,"props":1551,"children":1552},{},[1553,1562,1570,1578],{"type":45,"tag":178,"props":1554,"children":1555},{},[1556],{"type":45,"tag":78,"props":1557,"children":1559},{"className":1558},[],[1560],{"type":51,"value":1561},"n3_super",{"type":45,"tag":178,"props":1563,"children":1564},{},[1565],{"type":45,"tag":78,"props":1566,"children":1568},{"className":1567},[],[1569],{"type":51,"value":416},{"type":45,"tag":178,"props":1571,"children":1572},{},[1573],{"type":45,"tag":78,"props":1574,"children":1576},{"className":1575},[],[1577],{"type":51,"value":1145},{"type":45,"tag":178,"props":1579,"children":1580},{},[1581],{"type":51,"value":1582},"Nemotron3-Super on GB200; minimum supported size is 8 nodes",{"type":45,"tag":54,"props":1584,"children":1585},{},[1586],{"type":51,"value":1587},"Workload note:",{"type":45,"tag":1589,"props":1590,"children":1591},"ul",{},[1592,1603],{"type":45,"tag":1593,"props":1594,"children":1595},"li",{},[1596,1601],{"type":45,"tag":78,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":51,"value":950},{"type":51,"value":1602}," requires at least 2 nodes.",{"type":45,"tag":1593,"props":1604,"children":1605},{},[1606,1611],{"type":45,"tag":78,"props":1607,"children":1609},{"className":1608},[],[1610],{"type":51,"value":1561},{"type":51,"value":1612}," requires at least 8 nodes. Its default registered pool contains only 8-node experiments.",{"type":45,"tag":71,"props":1614,"children":1616},{"className":281,"code":1615,"language":283,"meta":80,"style":80},"# Run the full pool against the validated example workload\nbash scripts\u002Fprepare_node_alloc.sh\n\n# Run a custom subset against llama4_scout\nPOOL=\"GPU_SLEEP:1:5:2 SIGKILL:1:5:2\" WORKLOAD=llama4_scout bash scripts\u002Fprepare_node_alloc.sh\n",[1617],{"type":45,"tag":78,"props":1618,"children":1619},{"__ignoreMap":80},[1620,1629,1640,1649,1657],{"type":45,"tag":289,"props":1621,"children":1622},{"class":291,"line":292},[1623],{"type":45,"tag":289,"props":1624,"children":1626},{"style":1625},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1627],{"type":51,"value":1628},"# Run the full pool against the validated example workload\n",{"type":45,"tag":289,"props":1630,"children":1631},{"class":291,"line":450},[1632,1636],{"type":45,"tag":289,"props":1633,"children":1634},{"style":322},[1635],{"type":51,"value":283},{"type":45,"tag":289,"props":1637,"children":1638},{"style":312},[1639],{"type":51,"value":330},{"type":45,"tag":289,"props":1641,"children":1642},{"class":291,"line":488},[1643],{"type":45,"tag":289,"props":1644,"children":1646},{"emptyLinePlaceholder":1645},true,[1647],{"type":51,"value":1648},"\n",{"type":45,"tag":289,"props":1650,"children":1651},{"class":291,"line":522},[1652],{"type":45,"tag":289,"props":1653,"children":1654},{"style":1625},[1655],{"type":51,"value":1656},"# Run a custom subset against llama4_scout\n",{"type":45,"tag":289,"props":1658,"children":1659},{"class":291,"line":556},[1660,1664,1668,1672,1677,1681,1686,1690,1694,1698],{"type":45,"tag":289,"props":1661,"children":1662},{"style":296},[1663],{"type":51,"value":276},{"type":45,"tag":289,"props":1665,"children":1666},{"style":301},[1667],{"type":51,"value":304},{"type":45,"tag":289,"props":1669,"children":1670},{"style":301},[1671],{"type":51,"value":309},{"type":45,"tag":289,"props":1673,"children":1674},{"style":312},[1675],{"type":51,"value":1676},"GPU_SLEEP:1:5:2 SIGKILL:1:5:2",{"type":45,"tag":289,"props":1678,"children":1679},{"style":301},[1680],{"type":51,"value":309},{"type":45,"tag":289,"props":1682,"children":1683},{"style":296},[1684],{"type":51,"value":1685}," WORKLOAD",{"type":45,"tag":289,"props":1687,"children":1688},{"style":301},[1689],{"type":51,"value":304},{"type":45,"tag":289,"props":1691,"children":1692},{"style":312},[1693],{"type":51,"value":950},{"type":45,"tag":289,"props":1695,"children":1696},{"style":322},[1697],{"type":51,"value":325},{"type":45,"tag":289,"props":1699,"children":1700},{"style":312},[1701],{"type":51,"value":330},{"type":45,"tag":60,"props":1703,"children":1704},{},[],{"type":45,"tag":64,"props":1706,"children":1708},{"id":1707},"step-1-2-batched-submission-wait-automated",[1709],{"type":51,"value":1710},"Step 1 & 2 — Batched Submission + Wait (automated)",{"type":45,"tag":71,"props":1712,"children":1714},{"className":281,"code":1713,"language":283,"meta":80,"style":80},"bash scripts\u002Fprepare_node_alloc.sh\n",[1715],{"type":45,"tag":78,"props":1716,"children":1717},{"__ignoreMap":80},[1718],{"type":45,"tag":289,"props":1719,"children":1720},{"class":291,"line":292},[1721,1725],{"type":45,"tag":289,"props":1722,"children":1723},{"style":322},[1724],{"type":51,"value":283},{"type":45,"tag":289,"props":1726,"children":1727},{"style":312},[1728],{"type":51,"value":330},{"type":45,"tag":54,"props":1730,"children":1731},{},[1732,1734,1740],{"type":51,"value":1733},"The script loops: submit 2 jobs → poll ",{"type":45,"tag":78,"props":1735,"children":1737},{"className":1736},[],[1738],{"type":51,"value":1739},"squeue",{"type":51,"value":1741}," every 30 s until both finish →\nsubmit next 2. Progress is printed inline:",{"type":45,"tag":71,"props":1743,"children":1746},{"className":1744,"code":1745,"language":51},[74],">>> Batch 1: experiments 1–2 of 34\n  submitted: GPU_SLEEP rank=1  iter=5 nodes=2 -> job=1850\n  submitted: GPU_SLEEP rank=0  iter=5 nodes=2 -> job=1851\n  waiting for GPU_SLEEP:1:5:2 GPU_SLEEP:0:5:2 (1850,1851) ... 30s 60s done.\n>>> Batch 2: experiments 3–4 of 34\n  ...\n",[1747],{"type":45,"tag":78,"props":1748,"children":1749},{"__ignoreMap":80},[1750],{"type":51,"value":1745},{"type":45,"tag":54,"props":1752,"children":1753},{},[1754],{"type":51,"value":1755},"A session directory and TSV tracking file are created at launch time:",{"type":45,"tag":71,"props":1757,"children":1760},{"className":1758,"code":1759,"language":51},[74],"${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002F\u003CYYYYMMDD_HHMMSS>\u002F\n  experiments.tsv                              ← tracking file (all job IDs + paths)\n  n\u003CN>_\u003CFAULT>_r\u003CR>_i\u003CI>\u002F                     ← one subdir per experiment\n    logs\u002Fslurm\u002F\u003CJOB_ID>.launch.out\n    logs\u002Fslurm\u002F\u003CJOB_ID>.*.1.main_workload.log  ← log-analysis input\n    checkpoints\u002F                               ← fr-analysis input (FR dumps)\n    tensorboard\u002F\n  experiments_report.md                        ← generated by watch_and_analyze.sh\n",[1761],{"type":45,"tag":78,"props":1762,"children":1763},{"__ignoreMap":80},[1764],{"type":51,"value":1759},{"type":45,"tag":54,"props":1766,"children":1767},{},[1768,1770],{"type":51,"value":1769},"Tracking file columns: ",{"type":45,"tag":78,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":51,"value":1775},"JOB_ID  FAULT_TYPE  RANK  ITER  NODES  EXPERIMENT_DIR",{"type":45,"tag":60,"props":1777,"children":1778},{},[],{"type":45,"tag":64,"props":1780,"children":1782},{"id":1781},"step-3-analyze-all-experiments",[1783],{"type":51,"value":1784},"Step 3 — Analyze All Experiments",{"type":45,"tag":54,"props":1786,"children":1787},{},[1788],{"type":51,"value":1789},"Run the watcher\u002Fanalyzer — it reads the tracking file and processes each experiment\nas its job state leaves RUNNING\u002FPENDING (works whether jobs are still running or\nalready done):",{"type":45,"tag":71,"props":1791,"children":1793},{"className":281,"code":1792,"language":283,"meta":80,"style":80},"bash scripts\u002Fwatch_and_analyze.sh \\\n    ${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002F\u003CYYYYMMDD_HHMMSS>\u002Fexperiments.tsv\n",[1794],{"type":45,"tag":78,"props":1795,"children":1796},{"__ignoreMap":80},[1797,1814],{"type":45,"tag":289,"props":1798,"children":1799},{"class":291,"line":292},[1800,1804,1809],{"type":45,"tag":289,"props":1801,"children":1802},{"style":322},[1803],{"type":51,"value":283},{"type":45,"tag":289,"props":1805,"children":1806},{"style":312},[1807],{"type":51,"value":1808}," scripts\u002Fwatch_and_analyze.sh",{"type":45,"tag":289,"props":1810,"children":1811},{"style":296},[1812],{"type":51,"value":1813}," \\\n",{"type":45,"tag":289,"props":1815,"children":1816},{"class":291,"line":450},[1817,1822,1826,1830,1835,1840,1845,1850,1855],{"type":45,"tag":289,"props":1818,"children":1819},{"style":301},[1820],{"type":51,"value":1821},"    ${",{"type":45,"tag":289,"props":1823,"children":1824},{"style":296},[1825],{"type":51,"value":456},{"type":45,"tag":289,"props":1827,"children":1828},{"style":301},[1829],{"type":51,"value":475},{"type":45,"tag":289,"props":1831,"children":1832},{"style":312},[1833],{"type":51,"value":1834},"\u002Ffault_injection\u002F",{"type":45,"tag":289,"props":1836,"children":1837},{"style":301},[1838],{"type":51,"value":1839},"\u003C",{"type":45,"tag":289,"props":1841,"children":1842},{"style":312},[1843],{"type":51,"value":1844},"YYYYMMDD_HHMMS",{"type":45,"tag":289,"props":1846,"children":1847},{"style":296},[1848],{"type":51,"value":1849},"S",{"type":45,"tag":289,"props":1851,"children":1852},{"style":301},[1853],{"type":51,"value":1854},">",{"type":45,"tag":289,"props":1856,"children":1857},{"style":312},[1858],{"type":51,"value":1859},"\u002Fexperiments.tsv\n",{"type":45,"tag":54,"props":1861,"children":1862},{},[1863],{"type":51,"value":1864},"The watcher:",{"type":45,"tag":1866,"props":1867,"children":1868},"ol",{},[1869,1874,1901,1914,1919,1932],{"type":45,"tag":1593,"props":1870,"children":1871},{},[1872],{"type":51,"value":1873},"Reads each row from the tracking TSV",{"type":45,"tag":1593,"props":1875,"children":1876},{},[1877,1879,1885,1887,1893,1895],{"type":51,"value":1878},"Calls ",{"type":45,"tag":78,"props":1880,"children":1882},{"className":1881},[],[1883],{"type":51,"value":1884},"nvrx_logsage.py --exclude_nvrx_logs",{"type":51,"value":1886}," and parses the text output to get\n",{"type":45,"tag":78,"props":1888,"children":1890},{"className":1889},[],[1891],{"type":51,"value":1892},"restart_decision",{"type":51,"value":1894}," and ",{"type":45,"tag":78,"props":1896,"children":1898},{"className":1897},[],[1899],{"type":51,"value":1900},"attribution_text",{"type":45,"tag":1593,"props":1902,"children":1903},{},[1904,1906,1912],{"type":51,"value":1905},"Calls FR analysis as ",{"type":45,"tag":78,"props":1907,"children":1909},{"className":1908},[],[1910],{"type":51,"value":1911},"python -m nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution --fr-path \"${EXPERIMENT_DIR}\u002Fcheckpoints\" -p \"_dump_*\"",{"type":51,"value":1913}," and passes the raw table output to the judge",{"type":45,"tag":1593,"props":1915,"children":1916},{},[1917],{"type":51,"value":1918},"Scores 7 dimensions (restart correctness, rank primary, rank any, category, type, FR rank)",{"type":45,"tag":1593,"props":1920,"children":1921},{},[1922,1924,1930],{"type":51,"value":1923},"Appends a scored row to ",{"type":45,"tag":78,"props":1925,"children":1927},{"className":1926},[],[1928],{"type":51,"value":1929},"\u003Csession>_report.md",{"type":51,"value":1931}," as a markdown table row",{"type":45,"tag":1593,"props":1933,"children":1934},{},[1935],{"type":51,"value":1936},"Repeats until all experiments are analyzed",{"type":45,"tag":54,"props":1938,"children":1939},{},[1940],{"type":51,"value":1941},"To also run the sub-skills interactively for a single experiment:",{"type":45,"tag":71,"props":1943,"children":1945},{"className":281,"code":1944,"language":283,"meta":80,"style":80},"\u002Flog-analysis --log-path \"${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F${JOB_ID}.*.1.main_workload.log\"\n\u002Ffr-analysis  --fr-path \"${EXPERIMENT_DIR}\u002Fcheckpoints\" -p \"_dump_*\"\n",[1946],{"type":45,"tag":78,"props":1947,"children":1948},{"__ignoreMap":80},[1949,2004],{"type":45,"tag":289,"props":1950,"children":1951},{"class":291,"line":292},[1952,1957,1962,1967,1972,1976,1981,1986,1991,1995,2000],{"type":45,"tag":289,"props":1953,"children":1954},{"style":322},[1955],{"type":51,"value":1956},"\u002Flog-analysis",{"type":45,"tag":289,"props":1958,"children":1959},{"style":312},[1960],{"type":51,"value":1961}," --log-path",{"type":45,"tag":289,"props":1963,"children":1964},{"style":301},[1965],{"type":51,"value":1966}," \"${",{"type":45,"tag":289,"props":1968,"children":1969},{"style":296},[1970],{"type":51,"value":1971},"EXPERIMENT_DIR",{"type":45,"tag":289,"props":1973,"children":1974},{"style":301},[1975],{"type":51,"value":475},{"type":45,"tag":289,"props":1977,"children":1978},{"style":312},[1979],{"type":51,"value":1980},"\u002Flogs\u002Fslurm\u002F",{"type":45,"tag":289,"props":1982,"children":1983},{"style":301},[1984],{"type":51,"value":1985},"${",{"type":45,"tag":289,"props":1987,"children":1988},{"style":296},[1989],{"type":51,"value":1990},"JOB_ID",{"type":45,"tag":289,"props":1992,"children":1993},{"style":301},[1994],{"type":51,"value":475},{"type":45,"tag":289,"props":1996,"children":1997},{"style":312},[1998],{"type":51,"value":1999},".*.1.main_workload.log",{"type":45,"tag":289,"props":2001,"children":2002},{"style":301},[2003],{"type":51,"value":485},{"type":45,"tag":289,"props":2005,"children":2006},{"class":291,"line":450},[2007,2012,2017,2021,2025,2029,2034,2038,2043,2048,2053],{"type":45,"tag":289,"props":2008,"children":2009},{"style":322},[2010],{"type":51,"value":2011},"\u002Ffr-analysis",{"type":45,"tag":289,"props":2013,"children":2014},{"style":312},[2015],{"type":51,"value":2016},"  --fr-path",{"type":45,"tag":289,"props":2018,"children":2019},{"style":301},[2020],{"type":51,"value":1966},{"type":45,"tag":289,"props":2022,"children":2023},{"style":296},[2024],{"type":51,"value":1971},{"type":45,"tag":289,"props":2026,"children":2027},{"style":301},[2028],{"type":51,"value":475},{"type":45,"tag":289,"props":2030,"children":2031},{"style":312},[2032],{"type":51,"value":2033},"\u002Fcheckpoints",{"type":45,"tag":289,"props":2035,"children":2036},{"style":301},[2037],{"type":51,"value":309},{"type":45,"tag":289,"props":2039,"children":2040},{"style":312},[2041],{"type":51,"value":2042}," -p",{"type":45,"tag":289,"props":2044,"children":2045},{"style":301},[2046],{"type":51,"value":2047}," \"",{"type":45,"tag":289,"props":2049,"children":2050},{"style":312},[2051],{"type":51,"value":2052},"_dump_*",{"type":45,"tag":289,"props":2054,"children":2055},{"style":301},[2056],{"type":51,"value":485},{"type":45,"tag":60,"props":2058,"children":2059},{},[],{"type":45,"tag":64,"props":2061,"children":2063},{"id":2062},"step-4-score-each-experiment",[2064],{"type":51,"value":2065},"Step 4 — Score Each Experiment",{"type":45,"tag":54,"props":2067,"children":2068},{},[2069,2071,2077],{"type":51,"value":2070},"Scoring is performed by ",{"type":45,"tag":78,"props":2072,"children":2074},{"className":2073},[],[2075],{"type":51,"value":2076},"scripts\u002Fscore_attribution.py",{"type":51,"value":2078},", an LLM judge that\nreceives the ground truth, the filtered raw log, the logsage attribution output, and the FR\nanalysis output, then returns structured JSON scores with a reasoning note.",{"type":45,"tag":128,"props":2080,"children":2081},{},[2082,2103],{"type":45,"tag":132,"props":2083,"children":2084},{},[2085],{"type":45,"tag":136,"props":2086,"children":2087},{},[2088,2093,2098],{"type":45,"tag":140,"props":2089,"children":2090},{},[2091],{"type":51,"value":2092},"Column",{"type":45,"tag":140,"props":2094,"children":2095},{},[2096],{"type":51,"value":2097},"Values",{"type":45,"tag":140,"props":2099,"children":2100},{},[2101],{"type":51,"value":2102},"Meaning",{"type":45,"tag":171,"props":2104,"children":2105},{},[2106,2147,2184,2214,2250,2316],{"type":45,"tag":136,"props":2107,"children":2108},{},[2109,2118,2142],{"type":45,"tag":178,"props":2110,"children":2111},{},[2112],{"type":45,"tag":2113,"props":2114,"children":2115},"strong",{},[2116],{"type":51,"value":2117},"restart_correct",{"type":45,"tag":178,"props":2119,"children":2120},{},[2121,2127,2129,2135,2136],{"type":45,"tag":78,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":51,"value":2126},"true",{"type":51,"value":2128}," \u002F ",{"type":45,"tag":78,"props":2130,"children":2132},{"className":2131},[],[2133],{"type":51,"value":2134},"false",{"type":51,"value":2128},{"type":45,"tag":78,"props":2137,"children":2139},{"className":2138},[],[2140],{"type":51,"value":2141},"N\u002FA",{"type":45,"tag":178,"props":2143,"children":2144},{},[2145],{"type":51,"value":2146},"Restart decision matches expected for this fault type",{"type":45,"tag":136,"props":2148,"children":2149},{},[2150,2158,2179],{"type":45,"tag":178,"props":2151,"children":2152},{},[2153],{"type":45,"tag":2113,"props":2154,"children":2155},{},[2156],{"type":51,"value":2157},"rank_primary",{"type":45,"tag":178,"props":2159,"children":2160},{},[2161,2166,2167,2172,2173],{"type":45,"tag":78,"props":2162,"children":2164},{"className":2163},[],[2165],{"type":51,"value":2126},{"type":51,"value":2128},{"type":45,"tag":78,"props":2168,"children":2170},{"className":2169},[],[2171],{"type":51,"value":2134},{"type":51,"value":2128},{"type":45,"tag":78,"props":2174,"children":2176},{"className":2175},[],[2177],{"type":51,"value":2178},"partial",{"type":45,"tag":178,"props":2180,"children":2181},{},[2182],{"type":51,"value":2183},"Injected rank is the primary root-cause in attribution",{"type":45,"tag":136,"props":2185,"children":2186},{},[2187,2195,2209],{"type":45,"tag":178,"props":2188,"children":2189},{},[2190],{"type":45,"tag":2113,"props":2191,"children":2192},{},[2193],{"type":51,"value":2194},"rank_any",{"type":45,"tag":178,"props":2196,"children":2197},{},[2198,2203,2204],{"type":45,"tag":78,"props":2199,"children":2201},{"className":2200},[],[2202],{"type":51,"value":2126},{"type":51,"value":2128},{"type":45,"tag":78,"props":2205,"children":2207},{"className":2206},[],[2208],{"type":51,"value":2134},{"type":45,"tag":178,"props":2210,"children":2211},{},[2212],{"type":51,"value":2213},"Injected rank mentioned anywhere in attribution",{"type":45,"tag":136,"props":2215,"children":2216},{},[2217,2225,2245],{"type":45,"tag":178,"props":2218,"children":2219},{},[2220],{"type":45,"tag":2113,"props":2221,"children":2222},{},[2223],{"type":51,"value":2224},"fault_described",{"type":45,"tag":178,"props":2226,"children":2227},{},[2228,2233,2234,2239,2240],{"type":45,"tag":78,"props":2229,"children":2231},{"className":2230},[],[2232],{"type":51,"value":2126},{"type":51,"value":2128},{"type":45,"tag":78,"props":2235,"children":2237},{"className":2236},[],[2238],{"type":51,"value":2134},{"type":51,"value":2128},{"type":45,"tag":78,"props":2241,"children":2243},{"className":2242},[],[2244],{"type":51,"value":2178},{"type":45,"tag":178,"props":2246,"children":2247},{},[2248],{"type":51,"value":2249},"Fault nature (hang\u002Fcrash\u002Fsignal\u002Fexception) correctly described",{"type":45,"tag":136,"props":2251,"children":2252},{},[2253,2261,2297],{"type":45,"tag":178,"props":2254,"children":2255},{},[2256],{"type":45,"tag":2113,"props":2257,"children":2258},{},[2259],{"type":51,"value":2260},"fr_rank_correct",{"type":45,"tag":178,"props":2262,"children":2263},{},[2264,2270,2271,2277,2278,2284,2285,2290,2291],{"type":45,"tag":78,"props":2265,"children":2267},{"className":2266},[],[2268],{"type":51,"value":2269},"rank",{"type":51,"value":2128},{"type":45,"tag":78,"props":2272,"children":2274},{"className":2273},[],[2275],{"type":51,"value":2276},"node",{"type":51,"value":2128},{"type":45,"tag":78,"props":2279,"children":2281},{"className":2280},[],[2282],{"type":51,"value":2283},"segment",{"type":51,"value":2128},{"type":45,"tag":78,"props":2286,"children":2288},{"className":2287},[],[2289],{"type":51,"value":2134},{"type":51,"value":2128},{"type":45,"tag":78,"props":2292,"children":2294},{"className":2293},[],[2295],{"type":51,"value":2296},"no_dumps",{"type":45,"tag":178,"props":2298,"children":2299},{},[2300,2302,2307,2309,2314],{"type":51,"value":2301},"FR analysis narrows correctly to the injected rank, exactly one ",{"type":45,"tag":78,"props":2303,"children":2305},{"className":2304},[],[2306],{"type":51,"value":1036},{"type":51,"value":2308}," rank block containing that rank, the configured ",{"type":45,"tag":78,"props":2310,"children":2312},{"className":2311},[],[2313],{"type":51,"value":791},{"type":51,"value":2315}," rank block containing the injected rank, or fails to narrow usefully",{"type":45,"tag":136,"props":2317,"children":2318},{},[2319,2327,2332],{"type":45,"tag":178,"props":2320,"children":2321},{},[2322],{"type":45,"tag":2113,"props":2323,"children":2324},{},[2325],{"type":51,"value":2326},"judge_notes",{"type":45,"tag":178,"props":2328,"children":2329},{},[2330],{"type":51,"value":2331},"string",{"type":45,"tag":178,"props":2333,"children":2334},{},[2335],{"type":51,"value":2336},"One-sentence summary of the main gap or confirmation",{"type":45,"tag":54,"props":2338,"children":2339},{},[2340],{"type":51,"value":2341},"The judge is given:",{"type":45,"tag":1866,"props":2343,"children":2344},{},[2345,2376,2388,2401,2406,2417],{"type":45,"tag":1593,"props":2346,"children":2347},{},[2348,2350,2356,2357,2362,2363,2369,2370],{"type":51,"value":2349},"Ground truth: ",{"type":45,"tag":78,"props":2351,"children":2353},{"className":2352},[],[2354],{"type":51,"value":2355},"fault_type",{"type":51,"value":387},{"type":45,"tag":78,"props":2358,"children":2360},{"className":2359},[],[2361],{"type":51,"value":2269},{"type":51,"value":387},{"type":45,"tag":78,"props":2364,"children":2366},{"className":2365},[],[2367],{"type":51,"value":2368},"iter",{"type":51,"value":387},{"type":45,"tag":78,"props":2371,"children":2373},{"className":2372},[],[2374],{"type":51,"value":2375},"nodes",{"type":45,"tag":1593,"props":2377,"children":2378},{},[2379,2381,2387],{"type":51,"value":2380},"Expected restart decision + rationale (derived from ",{"type":45,"tag":78,"props":2382,"children":2384},{"className":2383},[],[2385],{"type":51,"value":2386},"score_attribution.py:_RESTART_TABLE",{"type":51,"value":963},{"type":45,"tag":1593,"props":2389,"children":2390},{},[2391,2393,2399],{"type":51,"value":2392},"Filtered raw log (last 400 lines, same ",{"type":45,"tag":78,"props":2394,"children":2396},{"className":2395},[],[2397],{"type":51,"value":2398},"exclude_nvrx_logs",{"type":51,"value":2400}," filtering as logsage)",{"type":45,"tag":1593,"props":2402,"children":2403},{},[2404],{"type":51,"value":2405},"Raw logsage stdout (5-field text format)",{"type":45,"tag":1593,"props":2407,"children":2408},{},[2409,2411],{"type":51,"value":2410},"Raw FR analysis table output from ",{"type":45,"tag":78,"props":2412,"children":2414},{"className":2413},[],[2415],{"type":51,"value":2416},"fr_attribution.py --fr-path ... -p \"_dump_*\"",{"type":45,"tag":1593,"props":2418,"children":2419},{},[2420,2425,2426,2431],{"type":45,"tag":78,"props":2421,"children":2423},{"className":2422},[],[2424],{"type":51,"value":1036},{"type":51,"value":1894},{"type":45,"tag":78,"props":2427,"children":2429},{"className":2428},[],[2430],{"type":51,"value":791},{"type":51,"value":2432}," to map the injected rank to exact node-sized and segment-sized scopes for FR scoring",{"type":45,"tag":54,"props":2434,"children":2435},{},[2436,2438,2443,2445,2451,2453,2459,2461,2466,2468,2473],{"type":51,"value":2437},"Default judge model: ",{"type":45,"tag":78,"props":2439,"children":2441},{"className":2440},[],[2442],{"type":51,"value":753},{"type":51,"value":2444},". Override with ",{"type":45,"tag":78,"props":2446,"children":2448},{"className":2447},[],[2449],{"type":51,"value":2450},"--model",{"type":51,"value":2452}," in ",{"type":45,"tag":78,"props":2454,"children":2456},{"className":2455},[],[2457],{"type":51,"value":2458},"score_attribution.py",{"type":51,"value":2460},".\nDefault segment size for FR scope scoring: ",{"type":45,"tag":78,"props":2462,"children":2464},{"className":2463},[],[2465],{"type":51,"value":248},{"type":51,"value":2467}," ranks. Override with ",{"type":45,"tag":78,"props":2469,"children":2471},{"className":2470},[],[2472],{"type":51,"value":791},{"type":51,"value":856},{"type":45,"tag":60,"props":2475,"children":2476},{},[],{"type":45,"tag":64,"props":2478,"children":2480},{"id":2479},"step-5-aggregate-results",[2481],{"type":51,"value":2482},"Step 5 — Aggregate Results",{"type":45,"tag":54,"props":2484,"children":2485},{},[2486,2488,2493],{"type":51,"value":2487},"The canonical output of the loop is the markdown table in ",{"type":45,"tag":78,"props":2489,"children":2491},{"className":2490},[],[2492],{"type":51,"value":1929},{"type":51,"value":2494},".\nWhen summarizing results for users, prefer linking to that file and reproducing the\nsame table shape rather than flattening the results into plain prose.",{"type":45,"tag":54,"props":2496,"children":2497},{},[2498,2500,2505],{"type":51,"value":2499},"The report markdown table from ",{"type":45,"tag":78,"props":2501,"children":2503},{"className":2502},[],[2504],{"type":51,"value":400},{"type":51,"value":2506}," gives a matrix view. Look for\npatterns across rows:",{"type":45,"tag":71,"props":2508,"children":2511},{"className":2509,"code":2510,"language":51},[74],"| FAULT_TYPE | NODES | RANK | restart_correct | rank_primary | rank_any | fault_described | fr_rank_correct | judge_notes |\n|------------|-------|------|-----------------|--------------|----------|-----------------|-----------------|-------------|\n| GPU_SLEEP  |   2   |  0   |      true       |    false     |   true   |      true       |      true       | rank-0 identified only in secondary issues |\n| GPU_SLEEP  |   2   |  1   |      true       |     true     |   true   |      true       |      true       | correct on all dimensions |\n| GPU_ERROR  |   2   |  1   |      false      |    false     |  false   |     partial     |      true       | LLM issued RESTART; rank not mentioned |\n| SIGKILL    |   2   |  0   |      true       |    false     |  false   |     false       |      true       | attribution describes timeout not kill signal |\n",[2512],{"type":45,"tag":78,"props":2513,"children":2514},{"__ignoreMap":80},[2515],{"type":51,"value":2510},{"type":45,"tag":54,"props":2517,"children":2518},{},[2519],{"type":51,"value":2520},"Common failure mode patterns and their meaning:",{"type":45,"tag":128,"props":2522,"children":2523},{},[2524,2540],{"type":45,"tag":132,"props":2525,"children":2526},{},[2527],{"type":45,"tag":136,"props":2528,"children":2529},{},[2530,2535],{"type":45,"tag":140,"props":2531,"children":2532},{},[2533],{"type":51,"value":2534},"Pattern",{"type":45,"tag":140,"props":2536,"children":2537},{},[2538],{"type":51,"value":2539},"Interpretation",{"type":45,"tag":171,"props":2541,"children":2542},{},[2543,2567,2586,2605,2624,2647,2671],{"type":45,"tag":136,"props":2544,"children":2545},{},[2546,2562],{"type":45,"tag":178,"props":2547,"children":2548},{},[2549,2555,2556],{"type":45,"tag":78,"props":2550,"children":2552},{"className":2551},[],[2553],{"type":51,"value":2554},"rank_primary=false",{"type":51,"value":387},{"type":45,"tag":78,"props":2557,"children":2559},{"className":2558},[],[2560],{"type":51,"value":2561},"rank_any=true",{"type":45,"tag":178,"props":2563,"children":2564},{},[2565],{"type":51,"value":2566},"Rank detected but treated as collateral; logsage putting it in secondary issues",{"type":45,"tag":136,"props":2568,"children":2569},{},[2570,2581],{"type":45,"tag":178,"props":2571,"children":2572},{},[2573,2579],{"type":45,"tag":78,"props":2574,"children":2576},{"className":2575},[],[2577],{"type":51,"value":2578},"rank_any=false",{"type":51,"value":2580}," for rank-0",{"type":45,"tag":178,"props":2582,"children":2583},{},[2584],{"type":51,"value":2585},"Rank-0 hang silences watchdog on other ranks; logsage lacks rank-0 signal",{"type":45,"tag":136,"props":2587,"children":2588},{},[2589,2600],{"type":45,"tag":178,"props":2590,"children":2591},{},[2592,2598],{"type":45,"tag":78,"props":2593,"children":2595},{"className":2594},[],[2596],{"type":51,"value":2597},"fault_described=partial",{"type":51,"value":2599}," for crash types",{"type":45,"tag":178,"props":2601,"children":2602},{},[2603],{"type":51,"value":2604},"Crash keywords present but fault type not specifically named",{"type":45,"tag":136,"props":2606,"children":2607},{},[2608,2619],{"type":45,"tag":178,"props":2609,"children":2610},{},[2611,2617],{"type":45,"tag":78,"props":2612,"children":2614},{"className":2613},[],[2615],{"type":51,"value":2616},"restart_correct=false",{"type":51,"value":2618}," for GPU_ERROR",{"type":45,"tag":178,"props":2620,"children":2621},{},[2622],{"type":51,"value":2623},"LLM conflating hardware error with recoverable hang",{"type":45,"tag":136,"props":2625,"children":2626},{},[2627,2636],{"type":45,"tag":178,"props":2628,"children":2629},{},[2630],{"type":45,"tag":78,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":51,"value":2635},"fr_rank_correct=no_dumps",{"type":45,"tag":178,"props":2637,"children":2638},{},[2639,2641],{"type":51,"value":2640},"NCCL watchdog did not fire before job ended — adjust ",{"type":45,"tag":78,"props":2642,"children":2644},{"className":2643},[],[2645],{"type":51,"value":2646},"TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC",{"type":45,"tag":136,"props":2648,"children":2649},{},[2650,2659],{"type":45,"tag":178,"props":2651,"children":2652},{},[2653],{"type":45,"tag":78,"props":2654,"children":2656},{"className":2655},[],[2657],{"type":51,"value":2658},"fr_rank_correct=node",{"type":45,"tag":178,"props":2660,"children":2661},{},[2662,2664,2669],{"type":51,"value":2663},"FR isolated exactly one ",{"type":45,"tag":78,"props":2665,"children":2667},{"className":2666},[],[2668],{"type":51,"value":1036},{"type":51,"value":2670}," rank block containing the injected rank, but not the exact rank",{"type":45,"tag":136,"props":2672,"children":2673},{},[2674,2683],{"type":45,"tag":178,"props":2675,"children":2676},{},[2677],{"type":45,"tag":78,"props":2678,"children":2680},{"className":2679},[],[2681],{"type":51,"value":2682},"fr_rank_correct=segment",{"type":45,"tag":178,"props":2684,"children":2685},{},[2686,2688,2693],{"type":51,"value":2687},"FR isolated the configured ",{"type":45,"tag":78,"props":2689,"children":2691},{"className":2690},[],[2692],{"type":51,"value":791},{"type":51,"value":2694}," rank block containing the injected rank, but not the exact node\u002Frank",{"type":45,"tag":60,"props":2696,"children":2697},{},[],{"type":45,"tag":64,"props":2699,"children":2701},{"id":2700},"step-6-identify-and-apply-improvements",[2702],{"type":51,"value":2703},"Step 6 — Identify and Apply Improvements",{"type":45,"tag":1475,"props":2705,"children":2706},{"id":40},[2707],{"type":51,"value":2708},"FR analysis",{"type":45,"tag":54,"props":2710,"children":2711},{},[2712,2714,2719],{"type":51,"value":2713},"Deterministic graph algorithm — ",{"type":45,"tag":2113,"props":2715,"children":2716},{},[2717],{"type":51,"value":2718},"do not modify automatically",{"type":51,"value":2720},".\nNote misidentifications and escalate to the team.",{"type":45,"tag":1475,"props":2722,"children":2724},{"id":2723},"log-analysis-safe-to-modify",[2725],{"type":51,"value":2726},"Log analysis (safe to modify)",{"type":45,"tag":128,"props":2728,"children":2729},{},[2730,2751],{"type":45,"tag":132,"props":2731,"children":2732},{},[2733],{"type":45,"tag":136,"props":2734,"children":2735},{},[2736,2741,2746],{"type":45,"tag":140,"props":2737,"children":2738},{},[2739],{"type":51,"value":2740},"Observation",{"type":45,"tag":140,"props":2742,"children":2743},{},[2744],{"type":51,"value":2745},"Target location",{"type":45,"tag":140,"props":2747,"children":2748},{},[2749],{"type":51,"value":2750},"Suggested fix",{"type":45,"tag":171,"props":2752,"children":2753},{},[2754,2786,2809,2830,2860,2878,2896],{"type":45,"tag":136,"props":2755,"children":2756},{},[2757,2762,2773],{"type":45,"tag":178,"props":2758,"children":2759},{},[2760],{"type":51,"value":2761},"Wrong restart for hang",{"type":45,"tag":178,"props":2763,"children":2764},{},[2765,2771],{"type":45,"tag":78,"props":2766,"children":2768},{"className":2767},[],[2769],{"type":51,"value":2770},"nvrx_logsage.py",{"type":51,"value":2772}," fast-path",{"type":45,"tag":178,"props":2774,"children":2775},{},[2776,2778,2784],{"type":51,"value":2777},"Strengthen NCCL timeout → ",{"type":45,"tag":78,"props":2779,"children":2781},{"className":2780},[],[2782],{"type":51,"value":2783},"RESTART IMMEDIATE",{"type":51,"value":2785}," mapping",{"type":45,"tag":136,"props":2787,"children":2788},{},[2789,2794,2804],{"type":45,"tag":178,"props":2790,"children":2791},{},[2792],{"type":51,"value":2793},"Missing rank in attr text",{"type":45,"tag":178,"props":2795,"children":2796},{},[2797,2802],{"type":45,"tag":78,"props":2798,"children":2800},{"className":2799},[],[2801],{"type":51,"value":2770},{"type":51,"value":2803}," prompt",{"type":45,"tag":178,"props":2805,"children":2806},{},[2807],{"type":51,"value":2808},"Extract rank from NCCL watchdog lines; add regex",{"type":45,"tag":136,"props":2810,"children":2811},{},[2812,2817,2825],{"type":45,"tag":178,"props":2813,"children":2814},{},[2815],{"type":51,"value":2816},"Crash misclassified as hang",{"type":45,"tag":178,"props":2818,"children":2819},{},[2820],{"type":45,"tag":78,"props":2821,"children":2823},{"className":2822},[],[2824],{"type":51,"value":2770},{"type":45,"tag":178,"props":2826,"children":2827},{},[2828],{"type":51,"value":2829},"Add SIGKILL\u002FSEGFAULT\u002FGPU_ERROR keyword patterns",{"type":45,"tag":136,"props":2831,"children":2832},{},[2833,2844,2855],{"type":45,"tag":178,"props":2834,"children":2835},{},[2836,2842],{"type":45,"tag":78,"props":2837,"children":2839},{"className":2838},[],[2840],{"type":51,"value":2841},"ERRORS NOT FOUND",{"type":51,"value":2843}," when errors exist",{"type":45,"tag":178,"props":2845,"children":2846},{},[2847,2853],{"type":45,"tag":78,"props":2848,"children":2850},{"className":2849},[],[2851],{"type":51,"value":2852},"return_application_errors",{"type":51,"value":2854}," config",{"type":45,"tag":178,"props":2856,"children":2857},{},[2858],{"type":51,"value":2859},"Loosen error extraction filter",{"type":45,"tag":136,"props":2861,"children":2862},{},[2863,2868,2873],{"type":45,"tag":178,"props":2864,"children":2865},{},[2866],{"type":51,"value":2867},"rank-0 not detected",{"type":45,"tag":178,"props":2869,"children":2870},{},[2871],{"type":51,"value":2872},"prompt or fast-path",{"type":45,"tag":178,"props":2874,"children":2875},{},[2876],{"type":51,"value":2877},"Add explicit rank-0 hang heuristic (other ranks silent)",{"type":45,"tag":136,"props":2879,"children":2880},{},[2881,2886,2891],{"type":45,"tag":178,"props":2882,"children":2883},{},[2884],{"type":51,"value":2885},"attr off by many iters",{"type":45,"tag":178,"props":2887,"children":2888},{},[2889],{"type":51,"value":2890},"prompt context",{"type":45,"tag":178,"props":2892,"children":2893},{},[2894],{"type":51,"value":2895},"Increase weight of iteration-stamped log lines",{"type":45,"tag":136,"props":2897,"children":2898},{},[2899,2904,2909],{"type":45,"tag":178,"props":2900,"children":2901},{},[2902],{"type":51,"value":2903},"LLM wrong on GPU_ERROR",{"type":45,"tag":178,"props":2905,"children":2906},{},[2907],{"type":51,"value":2908},"prompt",{"type":45,"tag":178,"props":2910,"children":2911},{},[2912,2914,2920],{"type":51,"value":2913},"Distinguish ",{"type":45,"tag":78,"props":2915,"children":2917},{"className":2916},[],[2918],{"type":51,"value":2919},"cudaError",{"type":51,"value":2921}," → crash from NCCL timeout → hang",{"type":45,"tag":54,"props":2923,"children":2924},{},[2925,2927],{"type":51,"value":2926},"Editable file: ",{"type":45,"tag":78,"props":2928,"children":2930},{"className":2929},[],[2931],{"type":51,"value":2932},"attribution\u002Flog_analyzer\u002Fnvrx_logsage.py",{"type":45,"tag":54,"props":2934,"children":2935},{},[2936],{"type":51,"value":2937},"After each patch, re-run the same pool subset that previously failed:",{"type":45,"tag":71,"props":2939,"children":2941},{"className":281,"code":2940,"language":283,"meta":80,"style":80},"POOL=\"GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2\" bash scripts\u002Fprepare_node_alloc.sh\n",[2942],{"type":45,"tag":78,"props":2943,"children":2944},{"__ignoreMap":80},[2945],{"type":45,"tag":289,"props":2946,"children":2947},{"class":291,"line":292},[2948,2952,2956,2960,2965,2969,2973],{"type":45,"tag":289,"props":2949,"children":2950},{"style":296},[2951],{"type":51,"value":276},{"type":45,"tag":289,"props":2953,"children":2954},{"style":301},[2955],{"type":51,"value":304},{"type":45,"tag":289,"props":2957,"children":2958},{"style":301},[2959],{"type":51,"value":309},{"type":45,"tag":289,"props":2961,"children":2962},{"style":312},[2963],{"type":51,"value":2964},"GPU_SLEEP:0:5:2 GPU_ERROR:1:5:2",{"type":45,"tag":289,"props":2966,"children":2967},{"style":301},[2968],{"type":51,"value":309},{"type":45,"tag":289,"props":2970,"children":2971},{"style":322},[2972],{"type":51,"value":325},{"type":45,"tag":289,"props":2974,"children":2975},{"style":312},[2976],{"type":51,"value":330},{"type":45,"tag":60,"props":2978,"children":2979},{},[],{"type":45,"tag":64,"props":2981,"children":2983},{"id":2982},"step-7-loop",[2984],{"type":51,"value":2985},"Step 7 — Loop",{"type":45,"tag":54,"props":2987,"children":2988},{},[2989],{"type":51,"value":2990},"Increment experiment counter. Suggested sweep order across code-change iterations:",{"type":45,"tag":1866,"props":2992,"children":2993},{},[2994,3004,3014,3024],{"type":45,"tag":1593,"props":2995,"children":2996},{},[2997,3002],{"type":45,"tag":2113,"props":2998,"children":2999},{},[3000],{"type":51,"value":3001},"Iteration 1",{"type":51,"value":3003},": full default pool (34 experiments)",{"type":45,"tag":1593,"props":3005,"children":3006},{},[3007,3012],{"type":45,"tag":2113,"props":3008,"children":3009},{},[3010],{"type":51,"value":3011},"Iteration 2",{"type":51,"value":3013},": targeted re-run of all failing cells from iteration 1",{"type":45,"tag":1593,"props":3015,"children":3016},{},[3017,3022],{"type":45,"tag":2113,"props":3018,"children":3019},{},[3020],{"type":51,"value":3021},"Iteration 3",{"type":51,"value":3023},": expand iter dimension (FAULT_AT_ITER=2 and 10) for remaining gaps",{"type":45,"tag":1593,"props":3025,"children":3026},{},[3027,3032],{"type":45,"tag":2113,"props":3028,"children":3029},{},[3030],{"type":51,"value":3031},"Iteration 4",{"type":51,"value":3033},": add SEGFAULT and LOCK_GIL 4-node\u002F8-node coverage",{"type":45,"tag":54,"props":3035,"children":3036},{},[3037],{"type":51,"value":3038},"Stop condition: all cells pass all four scoring dimensions for two consecutive\ncode-change iterations.",{"type":45,"tag":60,"props":3040,"children":3041},{},[],{"type":45,"tag":64,"props":3043,"children":3045},{"id":3044},"adapting-a-slurm-script-for-the-feedback-loop",[3046],{"type":51,"value":3047},"Adapting A SLURM Script For The Feedback Loop",{"type":45,"tag":54,"props":3049,"children":3050},{},[3051,3053,3058],{"type":51,"value":3052},"The feedback loop is not tied to ",{"type":45,"tag":78,"props":3054,"children":3056},{"className":3055},[],[3057],{"type":51,"value":408},{"type":51,"value":3059},", but your sbatch script must\nmatch a small contract so the loop can submit, analyze, and score each run.",{"type":45,"tag":54,"props":3061,"children":3062},{},[3063],{"type":51,"value":3064},"Required changes for a custom workload script:",{"type":45,"tag":1866,"props":3066,"children":3067},{},[3068,3118,3138,3150,3162],{"type":45,"tag":1593,"props":3069,"children":3070},{},[3071,3073,3078,3080,3085,3086,3092,3093,3098,3099,3104,3105,3110,3112,3117],{"type":51,"value":3072},"Accept these exported variables from ",{"type":45,"tag":78,"props":3074,"children":3076},{"className":3075},[],[3077],{"type":51,"value":393},{"type":51,"value":3079},":\n",{"type":45,"tag":78,"props":3081,"children":3083},{"className":3082},[],[3084],{"type":51,"value":839},{"type":51,"value":387},{"type":45,"tag":78,"props":3087,"children":3089},{"className":3088},[],[3090],{"type":51,"value":3091},"FAULT_RANK",{"type":51,"value":387},{"type":45,"tag":78,"props":3094,"children":3096},{"className":3095},[],[3097],{"type":51,"value":846},{"type":51,"value":387},{"type":45,"tag":78,"props":3100,"children":3102},{"className":3101},[],[3103],{"type":51,"value":1971},{"type":51,"value":387},{"type":45,"tag":78,"props":3106,"children":3108},{"className":3107},[],[3109],{"type":51,"value":456},{"type":51,"value":3111},",\nand ",{"type":45,"tag":78,"props":3113,"children":3115},{"className":3114},[],[3116],{"type":51,"value":1036},{"type":51,"value":856},{"type":45,"tag":1593,"props":3119,"children":3120},{},[3121,3123,3129,3131,3136],{"type":51,"value":3122},"Write the main training log to:\n",{"type":45,"tag":78,"props":3124,"children":3126},{"className":3125},[],[3127],{"type":51,"value":3128},"${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F${SLURM_JOB_ID}.*.1.main_workload.log",{"type":51,"value":3130},"\nso ",{"type":45,"tag":78,"props":3132,"children":3134},{"className":3133},[],[3135],{"type":51,"value":400},{"type":51,"value":3137}," can find it.",{"type":45,"tag":1593,"props":3139,"children":3140},{},[3141,3143,3149],{"type":51,"value":3142},"Write NCCL flight-recorder dumps under ",{"type":45,"tag":78,"props":3144,"children":3146},{"className":3145},[],[3147],{"type":51,"value":3148},"${EXPERIMENT_DIR}\u002Fcheckpoints\u002F",{"type":51,"value":856},{"type":45,"tag":1593,"props":3151,"children":3152},{},[3153,3155,3160],{"type":51,"value":3154},"Emit a fault-injection marker when the fault is injected.\n",{"type":45,"tag":78,"props":3156,"children":3158},{"className":3157},[],[3159],{"type":51,"value":400},{"type":51,"value":3161}," uses this to decide whether the run reached the injection point.",{"type":45,"tag":1593,"props":3163,"children":3164},{},[3165,3167,3173,3174,3180,3181,3187],{"type":51,"value":3166},"Preserve the per-experiment directory layout:\n",{"type":45,"tag":78,"props":3168,"children":3170},{"className":3169},[],[3171],{"type":51,"value":3172},"logs\u002Fslurm\u002F",{"type":51,"value":387},{"type":45,"tag":78,"props":3175,"children":3177},{"className":3176},[],[3178],{"type":51,"value":3179},"checkpoints\u002F",{"type":51,"value":410},{"type":45,"tag":78,"props":3182,"children":3184},{"className":3183},[],[3185],{"type":51,"value":3186},"tensorboard\u002F",{"type":51,"value":856},{"type":45,"tag":54,"props":3189,"children":3190},{},[3191,3193,3199,3201,3206],{"type":51,"value":3192},"This has only been validated with Megatron-LM because the current run-valid check and\nfault markers depend on Megatron's ",{"type":45,"tag":78,"props":3194,"children":3196},{"className":3195},[],[3197],{"type":51,"value":3198},"debug_fault_injection.py",{"type":51,"value":3200}," behavior. If you adapt the\nloop to another framework, update both the sbatch script and ",{"type":45,"tag":78,"props":3202,"children":3204},{"className":3203},[],[3205],{"type":51,"value":400},{"type":51,"value":856},{"type":45,"tag":64,"props":3208,"children":3210},{"id":3209},"appendix-a-sbatch_script-fault-parameters",[3211],{"type":51,"value":3212},"Appendix A: SBATCH_SCRIPT fault parameters",{"type":45,"tag":54,"props":3214,"children":3215},{},[3216,3218,3223,3225,3230,3232,3238],{"type":51,"value":3217},"The example ",{"type":45,"tag":78,"props":3219,"children":3221},{"className":3220},[],[3222],{"type":51,"value":1435},{"type":51,"value":3224}," reads these env vars from ",{"type":45,"tag":78,"props":3226,"children":3228},{"className":3227},[],[3229],{"type":51,"value":393},{"type":51,"value":3231}," via ",{"type":45,"tag":78,"props":3233,"children":3235},{"className":3234},[],[3236],{"type":51,"value":3237},"--export",{"type":51,"value":3239},":",{"type":45,"tag":128,"props":3241,"children":3242},{},[3243,3261],{"type":45,"tag":132,"props":3244,"children":3245},{},[3246],{"type":45,"tag":136,"props":3247,"children":3248},{},[3249,3253,3257],{"type":45,"tag":140,"props":3250,"children":3251},{},[3252],{"type":51,"value":917},{"type":45,"tag":140,"props":3254,"children":3255},{},[3256],{"type":51,"value":922},{"type":45,"tag":140,"props":3258,"children":3259},{},[3260],{"type":51,"value":927},{"type":45,"tag":171,"props":3262,"children":3263},{},[3264,3289,3313,3343,3368,3399,3424],{"type":45,"tag":136,"props":3265,"children":3266},{},[3267,3275,3284],{"type":45,"tag":178,"props":3268,"children":3269},{},[3270],{"type":45,"tag":78,"props":3271,"children":3273},{"className":3272},[],[3274],{"type":51,"value":846},{"type":45,"tag":178,"props":3276,"children":3277},{},[3278],{"type":45,"tag":78,"props":3279,"children":3281},{"className":3280},[],[3282],{"type":51,"value":3283},"5",{"type":45,"tag":178,"props":3285,"children":3286},{},[3287],{"type":51,"value":3288},"Training iteration at which to inject",{"type":45,"tag":136,"props":3290,"children":3291},{},[3292,3300,3308],{"type":45,"tag":178,"props":3293,"children":3294},{},[3295],{"type":45,"tag":78,"props":3296,"children":3298},{"className":3297},[],[3299],{"type":51,"value":854},{"type":45,"tag":178,"props":3301,"children":3302},{},[3303],{"type":45,"tag":78,"props":3304,"children":3306},{"className":3305},[],[3307],{"type":51,"value":236},{"type":45,"tag":178,"props":3309,"children":3310},{},[3311],{"type":51,"value":3312},"Delay in seconds before fault injection after the iteration anchor",{"type":45,"tag":136,"props":3314,"children":3315},{},[3316,3324,3332],{"type":45,"tag":178,"props":3317,"children":3318},{},[3319],{"type":45,"tag":78,"props":3320,"children":3322},{"className":3321},[],[3323],{"type":51,"value":3091},{"type":45,"tag":178,"props":3325,"children":3326},{},[3327],{"type":45,"tag":78,"props":3328,"children":3330},{"className":3329},[],[3331],{"type":51,"value":197},{"type":45,"tag":178,"props":3333,"children":3334},{},[3335,3337],{"type":51,"value":3336},"Global rank to inject ",{"type":45,"tag":78,"props":3338,"children":3340},{"className":3339},[],[3341],{"type":51,"value":3342},"[0, total_ranks)",{"type":45,"tag":136,"props":3344,"children":3345},{},[3346,3354,3363],{"type":45,"tag":178,"props":3347,"children":3348},{},[3349],{"type":45,"tag":78,"props":3350,"children":3352},{"className":3351},[],[3353],{"type":51,"value":839},{"type":45,"tag":178,"props":3355,"children":3356},{},[3357],{"type":45,"tag":78,"props":3358,"children":3360},{"className":3359},[],[3361],{"type":51,"value":3362},"GPU_SLEEP",{"type":45,"tag":178,"props":3364,"children":3365},{},[3366],{"type":51,"value":3367},"Megatron fault type enum name",{"type":45,"tag":136,"props":3369,"children":3370},{},[3371,3379,3387],{"type":45,"tag":178,"props":3372,"children":3373},{},[3374],{"type":45,"tag":78,"props":3375,"children":3377},{"className":3376},[],[3378],{"type":51,"value":1036},{"type":45,"tag":178,"props":3380,"children":3381},{},[3382],{"type":45,"tag":78,"props":3383,"children":3385},{"className":3384},[],[3386],{"type":51,"value":202},{"type":45,"tag":178,"props":3388,"children":3389},{},[3390,3392,3398],{"type":51,"value":3391},"GPUs per node (used to compute ",{"type":45,"tag":78,"props":3393,"children":3395},{"className":3394},[],[3396],{"type":51,"value":3397},"TOTAL_TASKS",{"type":51,"value":963},{"type":45,"tag":136,"props":3400,"children":3401},{},[3402,3410,3419],{"type":45,"tag":178,"props":3403,"children":3404},{},[3405],{"type":45,"tag":78,"props":3406,"children":3408},{"className":3407},[],[3409],{"type":51,"value":1971},{"type":45,"tag":178,"props":3411,"children":3412},{},[3413],{"type":45,"tag":78,"props":3414,"children":3416},{"className":3415},[],[3417],{"type":51,"value":3418},"${BASE_EXPERIMENTS_DIR}\u002Ffault_injection\u002Fn${SLURM_NNODES}_${FAULT_TYPE}_r${FAULT_RANK}_i${FAULT_AT_ITER}",{"type":45,"tag":178,"props":3420,"children":3421},{},[3422],{"type":51,"value":3423},"Per-experiment output root",{"type":45,"tag":136,"props":3425,"children":3426},{},[3427,3435,3443],{"type":45,"tag":178,"props":3428,"children":3429},{},[3430],{"type":45,"tag":78,"props":3431,"children":3433},{"className":3432},[],[3434],{"type":51,"value":456},{"type":45,"tag":178,"props":3436,"children":3437},{},[3438],{"type":45,"tag":78,"props":3439,"children":3441},{"className":3440},[],[3442],{"type":51,"value":1145},{"type":45,"tag":178,"props":3444,"children":3445},{},[3446],{"type":51,"value":3447},"Shared root (datacache, triton\u002Finductor caches)",{"type":45,"tag":54,"props":3449,"children":3450},{},[3451,3453,3458,3460,3466,3467,3472,3473,3479,3480,3486,3487,3493,3494,3500,3501,3507,3508,3514,3515,3521,3522,3528,3529,3535,3536],{"type":51,"value":3452},"Valid ",{"type":45,"tag":78,"props":3454,"children":3456},{"className":3455},[],[3457],{"type":51,"value":839},{"type":51,"value":3459}," values:\n",{"type":45,"tag":78,"props":3461,"children":3463},{"className":3462},[],[3464],{"type":51,"value":3465},"GPU_ERROR",{"type":51,"value":387},{"type":45,"tag":78,"props":3468,"children":3470},{"className":3469},[],[3471],{"type":51,"value":3362},{"type":51,"value":387},{"type":45,"tag":78,"props":3474,"children":3476},{"className":3475},[],[3477],{"type":51,"value":3478},"WORKLOAD_EXC",{"type":51,"value":387},{"type":45,"tag":78,"props":3481,"children":3483},{"className":3482},[],[3484],{"type":51,"value":3485},"ASYNC_EXC",{"type":51,"value":387},{"type":45,"tag":78,"props":3488,"children":3490},{"className":3489},[],[3491],{"type":51,"value":3492},"SIGNAL_EXC",{"type":51,"value":387},{"type":45,"tag":78,"props":3495,"children":3497},{"className":3496},[],[3498],{"type":51,"value":3499},"OS_ABORT",{"type":51,"value":402},{"type":45,"tag":78,"props":3502,"children":3504},{"className":3503},[],[3505],{"type":51,"value":3506},"LOCK_GIL",{"type":51,"value":387},{"type":45,"tag":78,"props":3509,"children":3511},{"className":3510},[],[3512],{"type":51,"value":3513},"SEGFAULT",{"type":51,"value":387},{"type":45,"tag":78,"props":3516,"children":3518},{"className":3517},[],[3519],{"type":51,"value":3520},"SIGINT",{"type":51,"value":387},{"type":45,"tag":78,"props":3523,"children":3525},{"className":3524},[],[3526],{"type":51,"value":3527},"SIGKILL",{"type":51,"value":387},{"type":45,"tag":78,"props":3530,"children":3532},{"className":3531},[],[3533],{"type":51,"value":3534},"SIGTERM",{"type":51,"value":387},{"type":45,"tag":78,"props":3537,"children":3539},{"className":3538},[],[3540],{"type":51,"value":3541},"SIGSTOP",{"type":45,"tag":60,"props":3543,"children":3544},{},[],{"type":45,"tag":64,"props":3546,"children":3548},{"id":3547},"appendix-b-single-experiment-manual-run",[3549],{"type":51,"value":3550},"Appendix B: Single-experiment manual run",{"type":45,"tag":71,"props":3552,"children":3554},{"className":281,"code":3553,"language":283,"meta":80,"style":80},"# Manual runs land under fault_injection\u002Fmanual\u002F by default (no session dir needed)\nEXPERIMENT_DIR=${HOME}\u002Fnvrx-attr-experiments\u002Ffault_injection\u002Fmanual\u002Fn2_GPU_SLEEP_r1_i5\nmkdir -p ${EXPERIMENT_DIR}\u002Flogs\u002Fslurm ${EXPERIMENT_DIR}\u002Fcheckpoints ${EXPERIMENT_DIR}\u002Ftensorboard\n\nsbatch \\\n    --nodes=2 \\\n    --output=${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F%j.launch.out \\\n    --error=${EXPERIMENT_DIR}\u002Flogs\u002Fslurm\u002F%j.launch.err \\\n    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=${EXPERIMENT_DIR} \\\n    scripts\u002Fl4_gb200_reduced.sh\n",[3555],{"type":45,"tag":78,"props":3556,"children":3557},{"__ignoreMap":80},[3558,3566,3591,3654,3661,3673,3685,3714,3743,3767],{"type":45,"tag":289,"props":3559,"children":3560},{"class":291,"line":292},[3561],{"type":45,"tag":289,"props":3562,"children":3563},{"style":1625},[3564],{"type":51,"value":3565},"# Manual runs land under fault_injection\u002Fmanual\u002F by default (no session dir needed)\n",{"type":45,"tag":289,"props":3567,"children":3568},{"class":291,"line":450},[3569,3573,3578,3582,3586],{"type":45,"tag":289,"props":3570,"children":3571},{"style":296},[3572],{"type":51,"value":1971},{"type":45,"tag":289,"props":3574,"children":3575},{"style":301},[3576],{"type":51,"value":3577},"=${",{"type":45,"tag":289,"props":3579,"children":3580},{"style":296},[3581],{"type":51,"value":470},{"type":45,"tag":289,"props":3583,"children":3584},{"style":301},[3585],{"type":51,"value":475},{"type":45,"tag":289,"props":3587,"children":3588},{"style":312},[3589],{"type":51,"value":3590},"\u002Fnvrx-attr-experiments\u002Ffault_injection\u002Fmanual\u002Fn2_GPU_SLEEP_r1_i5\n",{"type":45,"tag":289,"props":3592,"children":3593},{"class":291,"line":488},[3594,3599,3603,3608,3612,3616,3621,3625,3629,3633,3637,3641,3645,3649],{"type":45,"tag":289,"props":3595,"children":3596},{"style":322},[3597],{"type":51,"value":3598},"mkdir",{"type":45,"tag":289,"props":3600,"children":3601},{"style":312},[3602],{"type":51,"value":2042},{"type":45,"tag":289,"props":3604,"children":3605},{"style":301},[3606],{"type":51,"value":3607}," ${",{"type":45,"tag":289,"props":3609,"children":3610},{"style":296},[3611],{"type":51,"value":1971},{"type":45,"tag":289,"props":3613,"children":3614},{"style":301},[3615],{"type":51,"value":475},{"type":45,"tag":289,"props":3617,"children":3618},{"style":312},[3619],{"type":51,"value":3620},"\u002Flogs\u002Fslurm",{"type":45,"tag":289,"props":3622,"children":3623},{"style":301},[3624],{"type":51,"value":3607},{"type":45,"tag":289,"props":3626,"children":3627},{"style":296},[3628],{"type":51,"value":1971},{"type":45,"tag":289,"props":3630,"children":3631},{"style":301},[3632],{"type":51,"value":475},{"type":45,"tag":289,"props":3634,"children":3635},{"style":312},[3636],{"type":51,"value":2033},{"type":45,"tag":289,"props":3638,"children":3639},{"style":301},[3640],{"type":51,"value":3607},{"type":45,"tag":289,"props":3642,"children":3643},{"style":296},[3644],{"type":51,"value":1971},{"type":45,"tag":289,"props":3646,"children":3647},{"style":301},[3648],{"type":51,"value":475},{"type":45,"tag":289,"props":3650,"children":3651},{"style":312},[3652],{"type":51,"value":3653},"\u002Ftensorboard\n",{"type":45,"tag":289,"props":3655,"children":3656},{"class":291,"line":522},[3657],{"type":45,"tag":289,"props":3658,"children":3659},{"emptyLinePlaceholder":1645},[3660],{"type":51,"value":1648},{"type":45,"tag":289,"props":3662,"children":3663},{"class":291,"line":556},[3664,3669],{"type":45,"tag":289,"props":3665,"children":3666},{"style":322},[3667],{"type":51,"value":3668},"sbatch",{"type":45,"tag":289,"props":3670,"children":3671},{"style":296},[3672],{"type":51,"value":1813},{"type":45,"tag":289,"props":3674,"children":3675},{"class":291,"line":589},[3676,3681],{"type":45,"tag":289,"props":3677,"children":3678},{"style":312},[3679],{"type":51,"value":3680},"    --nodes=2",{"type":45,"tag":289,"props":3682,"children":3683},{"style":296},[3684],{"type":51,"value":1813},{"type":45,"tag":289,"props":3686,"children":3687},{"class":291,"line":615},[3688,3693,3697,3701,3705,3710],{"type":45,"tag":289,"props":3689,"children":3690},{"style":312},[3691],{"type":51,"value":3692},"    --output=",{"type":45,"tag":289,"props":3694,"children":3695},{"style":301},[3696],{"type":51,"value":1985},{"type":45,"tag":289,"props":3698,"children":3699},{"style":296},[3700],{"type":51,"value":1971},{"type":45,"tag":289,"props":3702,"children":3703},{"style":301},[3704],{"type":51,"value":475},{"type":45,"tag":289,"props":3706,"children":3707},{"style":312},[3708],{"type":51,"value":3709},"\u002Flogs\u002Fslurm\u002F%j.launch.out",{"type":45,"tag":289,"props":3711,"children":3712},{"style":296},[3713],{"type":51,"value":1813},{"type":45,"tag":289,"props":3715,"children":3716},{"class":291,"line":649},[3717,3722,3726,3730,3734,3739],{"type":45,"tag":289,"props":3718,"children":3719},{"style":312},[3720],{"type":51,"value":3721},"    --error=",{"type":45,"tag":289,"props":3723,"children":3724},{"style":301},[3725],{"type":51,"value":1985},{"type":45,"tag":289,"props":3727,"children":3728},{"style":296},[3729],{"type":51,"value":1971},{"type":45,"tag":289,"props":3731,"children":3732},{"style":301},[3733],{"type":51,"value":475},{"type":45,"tag":289,"props":3735,"children":3736},{"style":312},[3737],{"type":51,"value":3738},"\u002Flogs\u002Fslurm\u002F%j.launch.err",{"type":45,"tag":289,"props":3740,"children":3741},{"style":296},[3742],{"type":51,"value":1813},{"type":45,"tag":289,"props":3744,"children":3745},{"class":291,"line":682},[3746,3751,3755,3759,3763],{"type":45,"tag":289,"props":3747,"children":3748},{"style":312},[3749],{"type":51,"value":3750},"    --export=ALL,FAULT_TYPE=GPU_SLEEP,FAULT_RANK=1,FAULT_AT_ITER=5,FAULT_DELAY=15,GPUS_PER_NODE=4,EXPERIMENT_DIR=",{"type":45,"tag":289,"props":3752,"children":3753},{"style":301},[3754],{"type":51,"value":1985},{"type":45,"tag":289,"props":3756,"children":3757},{"style":296},[3758],{"type":51,"value":1971},{"type":45,"tag":289,"props":3760,"children":3761},{"style":301},[3762],{"type":51,"value":475},{"type":45,"tag":289,"props":3764,"children":3765},{"style":296},[3766],{"type":51,"value":1813},{"type":45,"tag":289,"props":3768,"children":3769},{"class":291,"line":708},[3770],{"type":45,"tag":289,"props":3771,"children":3772},{"style":312},[3773],{"type":51,"value":3774},"    scripts\u002Fl4_gb200_reduced.sh\n",{"type":45,"tag":54,"props":3776,"children":3777},{},[3778],{"type":51,"value":3779},"Optional site-specific cleanup:",{"type":45,"tag":71,"props":3781,"children":3783},{"className":281,"code":3782,"language":283,"meta":80,"style":80},"export CONTAINER_CLEANUP_CMD='\nENROOT_DIR=\"\u002Fvar\u002Flib\u002Fenroot\u002Fdata\u002F$(id -u)\"\nrm -rf \"${ENROOT_DIR:?}\"\u002F* 2>\u002Fdev\u002Fnull || true\necho \"$(hostname): \u002F $(df -h \u002F | tail -1 | awk \"{print \\$3\\\" used, \\\"\\$4\\\" avail\\\"}\")\"\n'\n",[3784],{"type":45,"tag":78,"props":3785,"children":3786},{"__ignoreMap":80},[3787,3810,3818,3826,3834],{"type":45,"tag":289,"props":3788,"children":3789},{"class":291,"line":292},[3790,3796,3801,3805],{"type":45,"tag":289,"props":3791,"children":3793},{"style":3792},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[3794],{"type":51,"value":3795},"export",{"type":45,"tag":289,"props":3797,"children":3798},{"style":296},[3799],{"type":51,"value":3800}," CONTAINER_CLEANUP_CMD",{"type":45,"tag":289,"props":3802,"children":3803},{"style":301},[3804],{"type":51,"value":304},{"type":45,"tag":289,"props":3806,"children":3807},{"style":301},[3808],{"type":51,"value":3809},"'\n",{"type":45,"tag":289,"props":3811,"children":3812},{"class":291,"line":450},[3813],{"type":45,"tag":289,"props":3814,"children":3815},{"style":312},[3816],{"type":51,"value":3817},"ENROOT_DIR=\"\u002Fvar\u002Flib\u002Fenroot\u002Fdata\u002F$(id -u)\"\n",{"type":45,"tag":289,"props":3819,"children":3820},{"class":291,"line":488},[3821],{"type":45,"tag":289,"props":3822,"children":3823},{"style":312},[3824],{"type":51,"value":3825},"rm -rf \"${ENROOT_DIR:?}\"\u002F* 2>\u002Fdev\u002Fnull || true\n",{"type":45,"tag":289,"props":3827,"children":3828},{"class":291,"line":522},[3829],{"type":45,"tag":289,"props":3830,"children":3831},{"style":312},[3832],{"type":51,"value":3833},"echo \"$(hostname): \u002F $(df -h \u002F | tail -1 | awk \"{print \\$3\\\" used, \\\"\\$4\\\" avail\\\"}\")\"\n",{"type":45,"tag":289,"props":3835,"children":3836},{"class":291,"line":556},[3837],{"type":45,"tag":289,"props":3838,"children":3839},{"style":301},[3840],{"type":51,"value":3809},{"type":45,"tag":3842,"props":3843,"children":3844},"style",{},[3845],{"type":51,"value":3846},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3848,"total":4005},[3849,3867,3885,3896,3908,3922,3935,3949,3960,3971,3985,3994],{"slug":3850,"name":3850,"fn":3851,"description":3852,"org":3853,"tags":3854,"stars":3864,"repoUrl":3865,"updatedAt":3866},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3855,3858,3861],{"name":3856,"slug":3857,"type":15},"Documentation","documentation",{"name":3859,"slug":3860,"type":15},"MCP","mcp",{"name":3862,"slug":3863,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":3868,"name":3868,"fn":3869,"description":3870,"org":3871,"tags":3872,"stars":3882,"repoUrl":3883,"updatedAt":3884},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3873,3876,3879],{"name":3874,"slug":3875,"type":15},"Containers","containers",{"name":3877,"slug":3878,"type":15},"Deployment","deployment",{"name":3880,"slug":3881,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":3886,"name":3886,"fn":3887,"description":3888,"org":3889,"tags":3890,"stars":3882,"repoUrl":3883,"updatedAt":3895},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3891,3894],{"name":3892,"slug":3893,"type":15},"CI\u002FCD","ci-cd",{"name":3877,"slug":3878,"type":15},"2026-07-14T05:25:59.97109",{"slug":3897,"name":3897,"fn":3898,"description":3899,"org":3900,"tags":3901,"stars":3882,"repoUrl":3883,"updatedAt":3907},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3902,3903,3904],{"name":3892,"slug":3893,"type":15},{"name":3877,"slug":3878,"type":15},{"name":3905,"slug":3906,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":3909,"name":3909,"fn":3910,"description":3911,"org":3912,"tags":3913,"stars":3882,"repoUrl":3883,"updatedAt":3921},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3914,3917,3918],{"name":3915,"slug":3916,"type":15},"Debugging","debugging",{"name":3905,"slug":3906,"type":15},{"name":3919,"slug":3920,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":3923,"name":3923,"fn":3924,"description":3925,"org":3926,"tags":3927,"stars":3882,"repoUrl":3883,"updatedAt":3934},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3928,3931],{"name":3929,"slug":3930,"type":15},"Best Practices","best-practices",{"name":3932,"slug":3933,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":3936,"name":3936,"fn":3937,"description":3938,"org":3939,"tags":3940,"stars":3882,"repoUrl":3883,"updatedAt":3948},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3941,3944,3947],{"name":3942,"slug":3943,"type":15},"Machine Learning","machine-learning",{"name":3945,"slug":3946,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":3950,"name":3950,"fn":3951,"description":3952,"org":3953,"tags":3954,"stars":3882,"repoUrl":3883,"updatedAt":3959},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3955,3958],{"name":3956,"slug":3957,"type":15},"QA","qa",{"name":21,"slug":22,"type":15},"2026-07-14T05:25:53.673039",{"slug":3961,"name":3961,"fn":3962,"description":3963,"org":3964,"tags":3965,"stars":3882,"repoUrl":3883,"updatedAt":3970},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3966,3967],{"name":3877,"slug":3878,"type":15},{"name":3968,"slug":3969,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":3972,"name":3972,"fn":3973,"description":3974,"org":3975,"tags":3976,"stars":3882,"repoUrl":3883,"updatedAt":3984},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3977,3980,3981],{"name":3978,"slug":3979,"type":15},"Code Review","code-review",{"name":3905,"slug":3906,"type":15},{"name":3982,"slug":3983,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":3986,"name":3986,"fn":3987,"description":3988,"org":3989,"tags":3990,"stars":3882,"repoUrl":3883,"updatedAt":3993},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3991,3992],{"name":3956,"slug":3957,"type":15},{"name":21,"slug":22,"type":15},"2026-07-14T05:25:54.928983",{"slug":3995,"name":3995,"fn":3996,"description":3997,"org":3998,"tags":3999,"stars":3882,"repoUrl":3883,"updatedAt":4004},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4000,4003],{"name":4001,"slug":4002,"type":15},"Automation","automation",{"name":3892,"slug":3893,"type":15},"2026-07-30T05:29:03.275638",496,{"items":4007,"total":522},[4008,4015,4029,4041],{"slug":4,"name":4,"fn":5,"description":6,"org":4009,"tags":4010,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4011,4012,4013,4014],{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},{"name":21,"slug":22,"type":15},{"slug":40,"name":40,"fn":4016,"description":4017,"org":4018,"tags":4019,"stars":23,"repoUrl":24,"updatedAt":4028},"analyze NCCL flight-recorder dumps","Analyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4020,4023,4024,4025],{"name":4021,"slug":4022,"type":15},"Analysis","analysis",{"name":3915,"slug":3916,"type":15},{"name":9,"slug":8,"type":15},{"name":4026,"slug":4027,"type":15},"Performance","performance","2026-07-14T05:35:13.703056",{"slug":39,"name":39,"fn":4030,"description":4031,"org":4032,"tags":4033,"stars":23,"repoUrl":24,"updatedAt":4040},"analyze SLURM training job logs","Analyze a SLURM job log file for failure root-cause attribution and restart decisions using NVRxLogAnalyzer. Use when you have a SLURM training job log and need to determine why the job failed and whether it should be restarted. Performs per-cycle chunking, fast-path pattern matching, and LLM-based classification.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4034,4035,4036,4039],{"name":4021,"slug":4022,"type":15},{"name":3915,"slug":3916,"type":15},{"name":4037,"slug":4038,"type":15},"Logs","logs",{"name":9,"slug":8,"type":15},"2026-07-14T05:35:14.94051",{"slug":4042,"name":4042,"fn":4043,"description":4044,"org":4045,"tags":4046,"stars":23,"repoUrl":24,"updatedAt":4053},"nvrx-attr","analyze attribution quality for Megatron-LM","Orchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4047,4050,4051,4052],{"name":4048,"slug":4049,"type":15},"AI Infrastructure","ai-infrastructure",{"name":13,"slug":14,"type":15},{"name":4037,"slug":4038,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:19.02075"]