[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-rl-auto-research":3,"mdc--4v7rht-key":34,"related-repo-nvidia-nemo-rl-auto-research":817,"related-org-nvidia-nemo-rl-auto-research":919},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-rl-auto-research","conduct autonomous NeMo-RL research experiments","Autonomous NeMo-RL research agent workflow for directed hypothesis testing and open-ended discovery. Guides agents through the full experiment lifecycle: understanding recipes and environments, wiring RL or NeMo-gym runs, launching reproducible baselines and iterations, analyzing results, preserving human oversight, and using git plus TSV logs as the research ledger. Do NOT use for: bug fixes, code review, documentation, refactoring, dependency updates, or single-file changes.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Research","research","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Agents","agents",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:44.014975","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-rl-auto-research","---\nname: nemo-rl-auto-research\nlicense: Apache-2.0\ndescription: \"Autonomous NeMo-RL research agent workflow for directed hypothesis testing and open-ended discovery. Guides agents through the full experiment lifecycle: understanding recipes and environments, wiring RL or NeMo-gym runs, launching reproducible baselines and iterations, analyzing results, preserving human oversight, and using git plus TSV logs as the research ledger. Do NOT use for: bug fixes, code review, documentation, refactoring, dependency updates, or single-file changes.\"\nwhen_to_use: auto research; run experiments; test these hypotheses; find a better recipe; improve accuracy; long-running NeMo-RL or NeMo-gym research campaigns; autonomous discovery; directed execution.\n---\n\n# Auto Research\n\nRun iterative NeMo-RL experiments in this repository against the user's stated objective, such as accuracy, reward, throughput, latency, stability, or another recipe-specific metric, with git as the research ledger.\n\nTreat dependencies as ready, but choose the runtime deliberately. Use the recipe's authoritative metric as the source of truth. Keep changes small, reproducible, and simple. Preserve unrelated user work.\n\n**Safety:** This skill creates git branches, writes files to disk, and executes shell commands including training jobs that may consume GPU resources. Always confirm the campaign plan with the user before creating branches or launching jobs. Do not execute destructive git operations (reset, force-push) or launch compute-intensive jobs without explicit user approval.\n\nUse the `nemo-rl-session-memory` skill for every auto-research campaign. Start or resume a session record before branching, then checkpoint after forming the plan, before and after meaningful edits or long-running launches, when the user changes direction, and before handoff or final summary.\n\nAfter context compaction, handoff, disconnect, or a long gap, reload this skill and any companion skills already in use, read the latest `nemo-rl-session-memory` handoff, and restate the overall objective, stop rules, current branch, and latest result before continuing. Treat follow-up steering as additive unless the user explicitly changes the main objective.\n\n## Workflow\n\n1. Inspect the current git state and identify unrelated user changes before branching.\n2. Use a shared branch prefix. Prefer a user-provided one; otherwise create a suggestive default such as `autoresearch\u002F2026-03-24-dapo-qwen2p5`.\n3. Read the target recipe, its parents, and the relevant code paths in `examples\u002Frun_grpo.py`, `nemo_rl\u002Fmodels\u002F`, `nemo_rl\u002Falgorithms\u002F`, `nemo_rl\u002Fenvironments\u002F`, and `docs\u002F`. For NeMo-gym recipes, also inspect `examples\u002Fnemo_gym\u002F` entrypoints, configs, and launch scripts.\n4. Translate any user stop rule into explicit values you can monitor, such as the requested number of experiments as `target_experiment_count`, `campaign_deadline`, `per_experiment_timeout`, or `target_metric`.\n5. Verify required data, checkpoints, runtime inputs, and the launcher.\n6. Create an untracked TSV log and per-experiment log directory.\n7. Run a baseline first on `\u003Cprefix>\u002Fbaseline` if none exists.\n\nFor GPU, CPU-heavy, distributed, or long-running work, choose the execution environment deliberately. Run locally when the current machine has suitable GPUs and capacity; otherwise follow the user's requested environment, use `launch-nemo-rl` for nrl-k8s\u002FKubernetes, use the environment's native launcher for Slurm, or clarify with the user before launching. Use CPU-only local runs only for light inspection, dry runs, and short non-GPU checks.\n\nIf the user mentions Brev, or if `\u002Fhome\u002Fubuntu\u002FRL` exists and `\u002Fephemeral` is available as a volume, treat the machine as a Brev instance and use `nemo-rl-brev-etiquette` before creating experiment directories, caches, logs, checkpoints, or authenticated runtime state.\n\n## Branching\n\n- Put every experiment on its own branch under the shared prefix.\n- Keep every branch, even for failed or weak ideas.\n- Put at least one commit on each branch for the hypothesis.\n- Add follow-up fix commits on the same branch when a rerun is justified.\n- Never stash, reset, or overwrite unrelated user changes silently. If dirty files overlap the experiment, use a separate worktree or ask before proceeding.\n\nSee `references\u002Fgit-workflow.md` for the exact pattern.\n\n## Loop\n\n1. Pick one concrete hypothesis.\n2. Create a branch such as `autoresearch\u002F2026-03-24-dapo-qwen2p5\u002Fprompt-compact-schema`.\n3. Edit the smallest set of files needed.\n4. Commit the hypothesis.\n5. Before launching the run, check the monitored stop conditions. Do not stop early unless one is already clearly met.\n6. Identify the authoritative metric source from the recipe or logging code, then run with a unique log path:\n\n```bash\nLOG_DIR=reports\u002Fauto_research\u002F\u003Ccampaign>\u002F\u003Cexperiment>\nmkdir -p \"$LOG_DIR\"\nuv run \u003Centrypoint> > \"$LOG_DIR\u002Frun.log\" 2>&1\n```\n\n7. If the user gave a per-experiment wall-clock limit, enforce it explicitly. Prefer a recipe-level timeout when one already exists; otherwise wrap the command with an external timeout. If both exist, honor the tighter limit.\n8. Extract the primary metric with a command appropriate for the actual log format. If extraction is empty, inspect the last log lines and the recipe's logging path before marking the run.\n9. Record index, branch, parent commit, commit, recipe, metric name, metric value, memory (GB), elapsed time (minutes), launcher, job id, command, log path, status, and description in the TSV, along with enough timing or count information to evaluate the stop rule.\n10. Periodically print user-facing progress updates during the campaign. Include the current branch, latest known result, attempted experiment count, remaining experiment count if applicable, remaining campaign time if applicable, and whether any stop condition has been met yet.\n11. Re-check the monitored stop conditions after the experiment completes and state the result explicitly, for example `stop condition not yet met: 17\u002F24 attempted, 6h12m remaining` or `stop condition met: 24\u002F24 attempted`.\n12. Mark the result as `keep`, `discard`, or `crash`, then move to the next branch unless a user-specified stop condition has been clearly met.\n\nFor count-based stop rules, count attempted ideas, not only successful or fully completed runs.\n\nFor campaign time budgets, convert the user limit into an absolute deadline at the start of the campaign and keep checking remaining time.\n\nFor per-experiment budgets, enforce a timeout on every run and treat overruns as failures.\n\nExamples:\n- `do 50 experiments`: stop only after 50 attempted experiment rows exist in the TSV\n- `10h total, 1h each`: enforce a 1 hour limit per run and stop when the 10 hour campaign budget is reached, or when there is not enough remaining budget to start another 1 hour run\n- `50 experiments or 10h total, 1h each`: monitor all three values, never exceed the per-run cap, and stop only when one campaign-level stop trigger is clearly reached\n\n## Priorities\n\nPrefer ideas with high expected objective gain and low complexity cost:\n- correctness and backend compatibility\n- prompt and rollout formatting\n- batch, sequence, and precision layout\n- optimizer and scheduler tuning\n- reward shaping, clipping, or scaling\n- dataset mix or validation changes\n- synchronous versus asynchronous execution based on hardware\n\nAll else equal, prefer simpler wins and avoid brittle hardware-specific hacks.\n\n## Avoid\n\n- Do not conclude a training idea failed from an underpowered smoke run. If a run uses tiny batch sizes, very few optimizer steps, or otherwise non-representative settings, treat it as plumbing validation only; scale to a meaningful batch size and train long enough to test the hypothesis before marking it `discard`.\n- Do not repeatedly pay batch-scheduler setup costs for tight edit-run-debug loops. If Slurm batch jobs have a large startup tax and failures require quick iteration, use the documented interactive Slurm pattern or ask the user before resubmitting more batch jobs.\n- Do not let context compaction or follow-up steering questions erase the original campaign goal. Refresh `nemo-rl-session-memory`, reload active skills, and preserve the main objective unless the user explicitly changes it.\n\n## Stop\n\nIf the user gives explicit stopping conditions, they override the generic rule. Do not stop because the search feels sufficient; stop only when the requested count, deadline, budget, or target condition has been clearly met.\n\nDuring the campaign, explicitly inform the user whether the stop condition has been met. If not, report the remaining count, remaining time, or other remaining threshold in concrete terms.\n\nIf the user does not give explicit stopping conditions, run the baseline plus up to three low-risk experiments, then summarize the best result and ask before continuing.\n\n## References\n\n- `references\u002Fgit-workflow.md` for branch, dirty-worktree, parent-commit, and baseline rules.\n- `references\u002Fexploration-ideas.md` for turning symptoms into concrete hypotheses.\n- `references\u002Fexperiment-log-template.md` for the TSV schema and reproducibility fields.\n",{"data":35,"body":37},{"name":4,"license":26,"description":6,"when_to_use":36},"auto research; run experiments; test these hypotheses; find a better recipe; improve accuracy; long-running NeMo-RL or NeMo-gym research campaigns; autonomous discovery; directed execution.",{"type":38,"children":39},"root",[40,49,55,60,71,85,97,104,235,248,277,283,312,325,331,371,531,602,607,612,617,622,658,664,669,707,712,718,749,755,760,765,770,776,811],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"auto-research",[46],{"type":47,"value":48},"text","Auto Research",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"Run iterative NeMo-RL experiments in this repository against the user's stated objective, such as accuracy, reward, throughput, latency, stability, or another recipe-specific metric, with git as the research ledger.",{"type":41,"tag":50,"props":56,"children":57},{},[58],{"type":47,"value":59},"Treat dependencies as ready, but choose the runtime deliberately. Use the recipe's authoritative metric as the source of truth. Keep changes small, reproducible, and simple. Preserve unrelated user work.",{"type":41,"tag":50,"props":61,"children":62},{},[63,69],{"type":41,"tag":64,"props":65,"children":66},"strong",{},[67],{"type":47,"value":68},"Safety:",{"type":47,"value":70}," This skill creates git branches, writes files to disk, and executes shell commands including training jobs that may consume GPU resources. Always confirm the campaign plan with the user before creating branches or launching jobs. Do not execute destructive git operations (reset, force-push) or launch compute-intensive jobs without explicit user approval.",{"type":41,"tag":50,"props":72,"children":73},{},[74,76,83],{"type":47,"value":75},"Use the ",{"type":41,"tag":77,"props":78,"children":80},"code",{"className":79},[],[81],{"type":47,"value":82},"nemo-rl-session-memory",{"type":47,"value":84}," skill for every auto-research campaign. Start or resume a session record before branching, then checkpoint after forming the plan, before and after meaningful edits or long-running launches, when the user changes direction, and before handoff or final summary.",{"type":41,"tag":50,"props":86,"children":87},{},[88,90,95],{"type":47,"value":89},"After context compaction, handoff, disconnect, or a long gap, reload this skill and any companion skills already in use, read the latest ",{"type":41,"tag":77,"props":91,"children":93},{"className":92},[],[94],{"type":47,"value":82},{"type":47,"value":96}," handoff, and restate the overall objective, stop rules, current branch, and latest result before continuing. Treat follow-up steering as additive unless the user explicitly changes the main objective.",{"type":41,"tag":98,"props":99,"children":101},"h2",{"id":100},"workflow",[102],{"type":47,"value":103},"Workflow",{"type":41,"tag":105,"props":106,"children":107},"ol",{},[108,114,127,178,212,217,222],{"type":41,"tag":109,"props":110,"children":111},"li",{},[112],{"type":47,"value":113},"Inspect the current git state and identify unrelated user changes before branching.",{"type":41,"tag":109,"props":115,"children":116},{},[117,119,125],{"type":47,"value":118},"Use a shared branch prefix. Prefer a user-provided one; otherwise create a suggestive default such as ",{"type":41,"tag":77,"props":120,"children":122},{"className":121},[],[123],{"type":47,"value":124},"autoresearch\u002F2026-03-24-dapo-qwen2p5",{"type":47,"value":126},".",{"type":41,"tag":109,"props":128,"children":129},{},[130,132,138,140,146,147,153,154,160,162,168,170,176],{"type":47,"value":131},"Read the target recipe, its parents, and the relevant code paths in ",{"type":41,"tag":77,"props":133,"children":135},{"className":134},[],[136],{"type":47,"value":137},"examples\u002Frun_grpo.py",{"type":47,"value":139},", ",{"type":41,"tag":77,"props":141,"children":143},{"className":142},[],[144],{"type":47,"value":145},"nemo_rl\u002Fmodels\u002F",{"type":47,"value":139},{"type":41,"tag":77,"props":148,"children":150},{"className":149},[],[151],{"type":47,"value":152},"nemo_rl\u002Falgorithms\u002F",{"type":47,"value":139},{"type":41,"tag":77,"props":155,"children":157},{"className":156},[],[158],{"type":47,"value":159},"nemo_rl\u002Fenvironments\u002F",{"type":47,"value":161},", and ",{"type":41,"tag":77,"props":163,"children":165},{"className":164},[],[166],{"type":47,"value":167},"docs\u002F",{"type":47,"value":169},". For NeMo-gym recipes, also inspect ",{"type":41,"tag":77,"props":171,"children":173},{"className":172},[],[174],{"type":47,"value":175},"examples\u002Fnemo_gym\u002F",{"type":47,"value":177}," entrypoints, configs, and launch scripts.",{"type":41,"tag":109,"props":179,"children":180},{},[181,183,189,190,196,197,203,205,211],{"type":47,"value":182},"Translate any user stop rule into explicit values you can monitor, such as the requested number of experiments as ",{"type":41,"tag":77,"props":184,"children":186},{"className":185},[],[187],{"type":47,"value":188},"target_experiment_count",{"type":47,"value":139},{"type":41,"tag":77,"props":191,"children":193},{"className":192},[],[194],{"type":47,"value":195},"campaign_deadline",{"type":47,"value":139},{"type":41,"tag":77,"props":198,"children":200},{"className":199},[],[201],{"type":47,"value":202},"per_experiment_timeout",{"type":47,"value":204},", or ",{"type":41,"tag":77,"props":206,"children":208},{"className":207},[],[209],{"type":47,"value":210},"target_metric",{"type":47,"value":126},{"type":41,"tag":109,"props":213,"children":214},{},[215],{"type":47,"value":216},"Verify required data, checkpoints, runtime inputs, and the launcher.",{"type":41,"tag":109,"props":218,"children":219},{},[220],{"type":47,"value":221},"Create an untracked TSV log and per-experiment log directory.",{"type":41,"tag":109,"props":223,"children":224},{},[225,227,233],{"type":47,"value":226},"Run a baseline first on ",{"type":41,"tag":77,"props":228,"children":230},{"className":229},[],[231],{"type":47,"value":232},"\u003Cprefix>\u002Fbaseline",{"type":47,"value":234}," if none exists.",{"type":41,"tag":50,"props":236,"children":237},{},[238,240,246],{"type":47,"value":239},"For GPU, CPU-heavy, distributed, or long-running work, choose the execution environment deliberately. Run locally when the current machine has suitable GPUs and capacity; otherwise follow the user's requested environment, use ",{"type":41,"tag":77,"props":241,"children":243},{"className":242},[],[244],{"type":47,"value":245},"launch-nemo-rl",{"type":47,"value":247}," for nrl-k8s\u002FKubernetes, use the environment's native launcher for Slurm, or clarify with the user before launching. Use CPU-only local runs only for light inspection, dry runs, and short non-GPU checks.",{"type":41,"tag":50,"props":249,"children":250},{},[251,253,259,261,267,269,275],{"type":47,"value":252},"If the user mentions Brev, or if ",{"type":41,"tag":77,"props":254,"children":256},{"className":255},[],[257],{"type":47,"value":258},"\u002Fhome\u002Fubuntu\u002FRL",{"type":47,"value":260}," exists and ",{"type":41,"tag":77,"props":262,"children":264},{"className":263},[],[265],{"type":47,"value":266},"\u002Fephemeral",{"type":47,"value":268}," is available as a volume, treat the machine as a Brev instance and use ",{"type":41,"tag":77,"props":270,"children":272},{"className":271},[],[273],{"type":47,"value":274},"nemo-rl-brev-etiquette",{"type":47,"value":276}," before creating experiment directories, caches, logs, checkpoints, or authenticated runtime state.",{"type":41,"tag":98,"props":278,"children":280},{"id":279},"branching",[281],{"type":47,"value":282},"Branching",{"type":41,"tag":284,"props":285,"children":286},"ul",{},[287,292,297,302,307],{"type":41,"tag":109,"props":288,"children":289},{},[290],{"type":47,"value":291},"Put every experiment on its own branch under the shared prefix.",{"type":41,"tag":109,"props":293,"children":294},{},[295],{"type":47,"value":296},"Keep every branch, even for failed or weak ideas.",{"type":41,"tag":109,"props":298,"children":299},{},[300],{"type":47,"value":301},"Put at least one commit on each branch for the hypothesis.",{"type":41,"tag":109,"props":303,"children":304},{},[305],{"type":47,"value":306},"Add follow-up fix commits on the same branch when a rerun is justified.",{"type":41,"tag":109,"props":308,"children":309},{},[310],{"type":47,"value":311},"Never stash, reset, or overwrite unrelated user changes silently. If dirty files overlap the experiment, use a separate worktree or ask before proceeding.",{"type":41,"tag":50,"props":313,"children":314},{},[315,317,323],{"type":47,"value":316},"See ",{"type":41,"tag":77,"props":318,"children":320},{"className":319},[],[321],{"type":47,"value":322},"references\u002Fgit-workflow.md",{"type":47,"value":324}," for the exact pattern.",{"type":41,"tag":98,"props":326,"children":328},{"id":327},"loop",[329],{"type":47,"value":330},"Loop",{"type":41,"tag":105,"props":332,"children":333},{},[334,339,351,356,361,366],{"type":41,"tag":109,"props":335,"children":336},{},[337],{"type":47,"value":338},"Pick one concrete hypothesis.",{"type":41,"tag":109,"props":340,"children":341},{},[342,344,350],{"type":47,"value":343},"Create a branch such as ",{"type":41,"tag":77,"props":345,"children":347},{"className":346},[],[348],{"type":47,"value":349},"autoresearch\u002F2026-03-24-dapo-qwen2p5\u002Fprompt-compact-schema",{"type":47,"value":126},{"type":41,"tag":109,"props":352,"children":353},{},[354],{"type":47,"value":355},"Edit the smallest set of files needed.",{"type":41,"tag":109,"props":357,"children":358},{},[359],{"type":47,"value":360},"Commit the hypothesis.",{"type":41,"tag":109,"props":362,"children":363},{},[364],{"type":47,"value":365},"Before launching the run, check the monitored stop conditions. Do not stop early unless one is already clearly met.",{"type":41,"tag":109,"props":367,"children":368},{},[369],{"type":47,"value":370},"Identify the authoritative metric source from the recipe or logging code, then run with a unique log path:",{"type":41,"tag":372,"props":373,"children":378},"pre",{"className":374,"code":375,"language":376,"meta":377,"style":377},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","LOG_DIR=reports\u002Fauto_research\u002F\u003Ccampaign>\u002F\u003Cexperiment>\nmkdir -p \"$LOG_DIR\"\nuv run \u003Centrypoint> > \"$LOG_DIR\u002Frun.log\" 2>&1\n","bash","",[379],{"type":41,"tag":77,"props":380,"children":381},{"__ignoreMap":377},[382,440,470],{"type":41,"tag":383,"props":384,"children":387},"span",{"class":385,"line":386},"line",1,[388,394,400,406,411,416,421,426,430,435],{"type":41,"tag":383,"props":389,"children":391},{"style":390},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[392],{"type":47,"value":393},"LOG_DIR",{"type":41,"tag":383,"props":395,"children":397},{"style":396},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[398],{"type":47,"value":399},"=",{"type":41,"tag":383,"props":401,"children":403},{"style":402},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[404],{"type":47,"value":405},"reports\u002Fauto_research\u002F",{"type":41,"tag":383,"props":407,"children":408},{"style":396},[409],{"type":47,"value":410},"\u003C",{"type":41,"tag":383,"props":412,"children":413},{"style":402},[414],{"type":47,"value":415},"campaign",{"type":41,"tag":383,"props":417,"children":418},{"style":396},[419],{"type":47,"value":420},">",{"type":41,"tag":383,"props":422,"children":423},{"style":402},[424],{"type":47,"value":425},"\u002F",{"type":41,"tag":383,"props":427,"children":428},{"style":396},[429],{"type":47,"value":410},{"type":41,"tag":383,"props":431,"children":432},{"style":402},[433],{"type":47,"value":434},"experiment",{"type":41,"tag":383,"props":436,"children":437},{"style":396},[438],{"type":47,"value":439},">\n",{"type":41,"tag":383,"props":441,"children":443},{"class":385,"line":442},2,[444,450,455,460,465],{"type":41,"tag":383,"props":445,"children":447},{"style":446},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[448],{"type":47,"value":449},"mkdir",{"type":41,"tag":383,"props":451,"children":452},{"style":402},[453],{"type":47,"value":454}," -p",{"type":41,"tag":383,"props":456,"children":457},{"style":396},[458],{"type":47,"value":459}," \"",{"type":41,"tag":383,"props":461,"children":462},{"style":390},[463],{"type":47,"value":464},"$LOG_DIR",{"type":41,"tag":383,"props":466,"children":467},{"style":396},[468],{"type":47,"value":469},"\"\n",{"type":41,"tag":383,"props":471,"children":473},{"class":385,"line":472},3,[474,479,484,489,494,499,503,508,512,516,521,526],{"type":41,"tag":383,"props":475,"children":476},{"style":446},[477],{"type":47,"value":478},"uv",{"type":41,"tag":383,"props":480,"children":481},{"style":402},[482],{"type":47,"value":483}," run",{"type":41,"tag":383,"props":485,"children":486},{"style":396},[487],{"type":47,"value":488}," \u003C",{"type":41,"tag":383,"props":490,"children":491},{"style":402},[492],{"type":47,"value":493},"entrypoin",{"type":41,"tag":383,"props":495,"children":496},{"style":390},[497],{"type":47,"value":498},"t",{"type":41,"tag":383,"props":500,"children":501},{"style":396},[502],{"type":47,"value":420},{"type":41,"tag":383,"props":504,"children":505},{"style":396},[506],{"type":47,"value":507}," >",{"type":41,"tag":383,"props":509,"children":510},{"style":396},[511],{"type":47,"value":459},{"type":41,"tag":383,"props":513,"children":514},{"style":390},[515],{"type":47,"value":464},{"type":41,"tag":383,"props":517,"children":518},{"style":402},[519],{"type":47,"value":520},"\u002Frun.log",{"type":41,"tag":383,"props":522,"children":523},{"style":396},[524],{"type":47,"value":525},"\"",{"type":41,"tag":383,"props":527,"children":528},{"style":396},[529],{"type":47,"value":530}," 2>&1\n",{"type":41,"tag":105,"props":532,"children":534},{"start":533},7,[535,540,545,550,555,575],{"type":41,"tag":109,"props":536,"children":537},{},[538],{"type":47,"value":539},"If the user gave a per-experiment wall-clock limit, enforce it explicitly. Prefer a recipe-level timeout when one already exists; otherwise wrap the command with an external timeout. If both exist, honor the tighter limit.",{"type":41,"tag":109,"props":541,"children":542},{},[543],{"type":47,"value":544},"Extract the primary metric with a command appropriate for the actual log format. If extraction is empty, inspect the last log lines and the recipe's logging path before marking the run.",{"type":41,"tag":109,"props":546,"children":547},{},[548],{"type":47,"value":549},"Record index, branch, parent commit, commit, recipe, metric name, metric value, memory (GB), elapsed time (minutes), launcher, job id, command, log path, status, and description in the TSV, along with enough timing or count information to evaluate the stop rule.",{"type":41,"tag":109,"props":551,"children":552},{},[553],{"type":47,"value":554},"Periodically print user-facing progress updates during the campaign. Include the current branch, latest known result, attempted experiment count, remaining experiment count if applicable, remaining campaign time if applicable, and whether any stop condition has been met yet.",{"type":41,"tag":109,"props":556,"children":557},{},[558,560,566,568,574],{"type":47,"value":559},"Re-check the monitored stop conditions after the experiment completes and state the result explicitly, for example ",{"type":41,"tag":77,"props":561,"children":563},{"className":562},[],[564],{"type":47,"value":565},"stop condition not yet met: 17\u002F24 attempted, 6h12m remaining",{"type":47,"value":567}," or ",{"type":41,"tag":77,"props":569,"children":571},{"className":570},[],[572],{"type":47,"value":573},"stop condition met: 24\u002F24 attempted",{"type":47,"value":126},{"type":41,"tag":109,"props":576,"children":577},{},[578,580,586,587,593,594,600],{"type":47,"value":579},"Mark the result as ",{"type":41,"tag":77,"props":581,"children":583},{"className":582},[],[584],{"type":47,"value":585},"keep",{"type":47,"value":139},{"type":41,"tag":77,"props":588,"children":590},{"className":589},[],[591],{"type":47,"value":592},"discard",{"type":47,"value":204},{"type":41,"tag":77,"props":595,"children":597},{"className":596},[],[598],{"type":47,"value":599},"crash",{"type":47,"value":601},", then move to the next branch unless a user-specified stop condition has been clearly met.",{"type":41,"tag":50,"props":603,"children":604},{},[605],{"type":47,"value":606},"For count-based stop rules, count attempted ideas, not only successful or fully completed runs.",{"type":41,"tag":50,"props":608,"children":609},{},[610],{"type":47,"value":611},"For campaign time budgets, convert the user limit into an absolute deadline at the start of the campaign and keep checking remaining time.",{"type":41,"tag":50,"props":613,"children":614},{},[615],{"type":47,"value":616},"For per-experiment budgets, enforce a timeout on every run and treat overruns as failures.",{"type":41,"tag":50,"props":618,"children":619},{},[620],{"type":47,"value":621},"Examples:",{"type":41,"tag":284,"props":623,"children":624},{},[625,636,647],{"type":41,"tag":109,"props":626,"children":627},{},[628,634],{"type":41,"tag":77,"props":629,"children":631},{"className":630},[],[632],{"type":47,"value":633},"do 50 experiments",{"type":47,"value":635},": stop only after 50 attempted experiment rows exist in the TSV",{"type":41,"tag":109,"props":637,"children":638},{},[639,645],{"type":41,"tag":77,"props":640,"children":642},{"className":641},[],[643],{"type":47,"value":644},"10h total, 1h each",{"type":47,"value":646},": enforce a 1 hour limit per run and stop when the 10 hour campaign budget is reached, or when there is not enough remaining budget to start another 1 hour run",{"type":41,"tag":109,"props":648,"children":649},{},[650,656],{"type":41,"tag":77,"props":651,"children":653},{"className":652},[],[654],{"type":47,"value":655},"50 experiments or 10h total, 1h each",{"type":47,"value":657},": monitor all three values, never exceed the per-run cap, and stop only when one campaign-level stop trigger is clearly reached",{"type":41,"tag":98,"props":659,"children":661},{"id":660},"priorities",[662],{"type":47,"value":663},"Priorities",{"type":41,"tag":50,"props":665,"children":666},{},[667],{"type":47,"value":668},"Prefer ideas with high expected objective gain and low complexity cost:",{"type":41,"tag":284,"props":670,"children":671},{},[672,677,682,687,692,697,702],{"type":41,"tag":109,"props":673,"children":674},{},[675],{"type":47,"value":676},"correctness and backend compatibility",{"type":41,"tag":109,"props":678,"children":679},{},[680],{"type":47,"value":681},"prompt and rollout formatting",{"type":41,"tag":109,"props":683,"children":684},{},[685],{"type":47,"value":686},"batch, sequence, and precision layout",{"type":41,"tag":109,"props":688,"children":689},{},[690],{"type":47,"value":691},"optimizer and scheduler tuning",{"type":41,"tag":109,"props":693,"children":694},{},[695],{"type":47,"value":696},"reward shaping, clipping, or scaling",{"type":41,"tag":109,"props":698,"children":699},{},[700],{"type":47,"value":701},"dataset mix or validation changes",{"type":41,"tag":109,"props":703,"children":704},{},[705],{"type":47,"value":706},"synchronous versus asynchronous execution based on hardware",{"type":41,"tag":50,"props":708,"children":709},{},[710],{"type":47,"value":711},"All else equal, prefer simpler wins and avoid brittle hardware-specific hacks.",{"type":41,"tag":98,"props":713,"children":715},{"id":714},"avoid",[716],{"type":47,"value":717},"Avoid",{"type":41,"tag":284,"props":719,"children":720},{},[721,732,737],{"type":41,"tag":109,"props":722,"children":723},{},[724,726,731],{"type":47,"value":725},"Do not conclude a training idea failed from an underpowered smoke run. If a run uses tiny batch sizes, very few optimizer steps, or otherwise non-representative settings, treat it as plumbing validation only; scale to a meaningful batch size and train long enough to test the hypothesis before marking it ",{"type":41,"tag":77,"props":727,"children":729},{"className":728},[],[730],{"type":47,"value":592},{"type":47,"value":126},{"type":41,"tag":109,"props":733,"children":734},{},[735],{"type":47,"value":736},"Do not repeatedly pay batch-scheduler setup costs for tight edit-run-debug loops. If Slurm batch jobs have a large startup tax and failures require quick iteration, use the documented interactive Slurm pattern or ask the user before resubmitting more batch jobs.",{"type":41,"tag":109,"props":738,"children":739},{},[740,742,747],{"type":47,"value":741},"Do not let context compaction or follow-up steering questions erase the original campaign goal. Refresh ",{"type":41,"tag":77,"props":743,"children":745},{"className":744},[],[746],{"type":47,"value":82},{"type":47,"value":748},", reload active skills, and preserve the main objective unless the user explicitly changes it.",{"type":41,"tag":98,"props":750,"children":752},{"id":751},"stop",[753],{"type":47,"value":754},"Stop",{"type":41,"tag":50,"props":756,"children":757},{},[758],{"type":47,"value":759},"If the user gives explicit stopping conditions, they override the generic rule. Do not stop because the search feels sufficient; stop only when the requested count, deadline, budget, or target condition has been clearly met.",{"type":41,"tag":50,"props":761,"children":762},{},[763],{"type":47,"value":764},"During the campaign, explicitly inform the user whether the stop condition has been met. If not, report the remaining count, remaining time, or other remaining threshold in concrete terms.",{"type":41,"tag":50,"props":766,"children":767},{},[768],{"type":47,"value":769},"If the user does not give explicit stopping conditions, run the baseline plus up to three low-risk experiments, then summarize the best result and ask before continuing.",{"type":41,"tag":98,"props":771,"children":773},{"id":772},"references",[774],{"type":47,"value":775},"References",{"type":41,"tag":284,"props":777,"children":778},{},[779,789,800],{"type":41,"tag":109,"props":780,"children":781},{},[782,787],{"type":41,"tag":77,"props":783,"children":785},{"className":784},[],[786],{"type":47,"value":322},{"type":47,"value":788}," for branch, dirty-worktree, parent-commit, and baseline rules.",{"type":41,"tag":109,"props":790,"children":791},{},[792,798],{"type":41,"tag":77,"props":793,"children":795},{"className":794},[],[796],{"type":47,"value":797},"references\u002Fexploration-ideas.md",{"type":47,"value":799}," for turning symptoms into concrete hypotheses.",{"type":41,"tag":109,"props":801,"children":802},{},[803,809],{"type":41,"tag":77,"props":804,"children":806},{"className":805},[],[807],{"type":47,"value":808},"references\u002Fexperiment-log-template.md",{"type":47,"value":810}," for the TSV schema and reproducibility fields.",{"type":41,"tag":812,"props":813,"children":814},"style",{},[815],{"type":47,"value":816},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":818,"total":918},[819,836,850,860,872,889,904],{"slug":820,"name":820,"fn":821,"description":822,"org":823,"tags":824,"stars":23,"repoUrl":24,"updatedAt":835},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[825,828,831,832],{"name":826,"slug":827,"type":15},"Data Analysis","data-analysis",{"name":829,"slug":830,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":833,"slug":834,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":837,"name":837,"fn":838,"description":839,"org":840,"tags":841,"stars":23,"repoUrl":24,"updatedAt":849},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[842,845,848],{"name":843,"slug":844,"type":15},"Deployment","deployment",{"name":846,"slug":847,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":851,"name":851,"fn":852,"description":853,"org":854,"tags":855,"stars":23,"repoUrl":24,"updatedAt":859},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[856,857,858],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:06.816956",{"slug":861,"name":861,"fn":862,"description":863,"org":864,"tags":865,"stars":23,"repoUrl":24,"updatedAt":871},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[866,867,868],{"name":826,"slug":827,"type":15},{"name":9,"slug":8,"type":15},{"name":869,"slug":870,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":873,"name":873,"fn":874,"description":875,"org":876,"tags":877,"stars":23,"repoUrl":24,"updatedAt":888},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[878,881,884,885],{"name":879,"slug":880,"type":15},"Automation","automation",{"name":882,"slug":883,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":886,"slug":887,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":890,"name":890,"fn":891,"description":892,"org":893,"tags":894,"stars":23,"repoUrl":24,"updatedAt":903},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[895,896,899,900],{"name":843,"slug":844,"type":15},{"name":897,"slug":898,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":901,"slug":902,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":905,"name":905,"fn":906,"description":907,"org":908,"tags":909,"stars":23,"repoUrl":24,"updatedAt":917},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[910,911,914],{"name":9,"slug":8,"type":15},{"name":912,"slug":913,"type":15},"Quantum Computing","quantum-computing",{"name":915,"slug":916,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":920,"total":1071},[921,939,955,966,978,992,1005,1019,1030,1039,1053,1062],{"slug":922,"name":922,"fn":923,"description":924,"org":925,"tags":926,"stars":936,"repoUrl":937,"updatedAt":938},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[927,930,933],{"name":928,"slug":929,"type":15},"Documentation","documentation",{"name":931,"slug":932,"type":15},"MCP","mcp",{"name":934,"slug":935,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":940,"name":940,"fn":941,"description":942,"org":943,"tags":944,"stars":952,"repoUrl":953,"updatedAt":954},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[945,948,949],{"name":946,"slug":947,"type":15},"Containers","containers",{"name":843,"slug":844,"type":15},{"name":950,"slug":951,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":956,"name":956,"fn":957,"description":958,"org":959,"tags":960,"stars":952,"repoUrl":953,"updatedAt":965},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[961,964],{"name":962,"slug":963,"type":15},"CI\u002FCD","ci-cd",{"name":843,"slug":844,"type":15},"2026-07-14T05:25:59.97109",{"slug":967,"name":967,"fn":968,"description":969,"org":970,"tags":971,"stars":952,"repoUrl":953,"updatedAt":977},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[972,973,974],{"name":962,"slug":963,"type":15},{"name":843,"slug":844,"type":15},{"name":975,"slug":976,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":979,"name":979,"fn":980,"description":981,"org":982,"tags":983,"stars":952,"repoUrl":953,"updatedAt":991},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[984,987,988],{"name":985,"slug":986,"type":15},"Debugging","debugging",{"name":975,"slug":976,"type":15},{"name":989,"slug":990,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":993,"name":993,"fn":994,"description":995,"org":996,"tags":997,"stars":952,"repoUrl":953,"updatedAt":1004},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[998,1001],{"name":999,"slug":1000,"type":15},"Best Practices","best-practices",{"name":1002,"slug":1003,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1006,"name":1006,"fn":1007,"description":1008,"org":1009,"tags":1010,"stars":952,"repoUrl":953,"updatedAt":1018},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1011,1014,1017],{"name":1012,"slug":1013,"type":15},"Machine Learning","machine-learning",{"name":1015,"slug":1016,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1020,"name":1020,"fn":1021,"description":1022,"org":1023,"tags":1024,"stars":952,"repoUrl":953,"updatedAt":1029},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1025,1028],{"name":1026,"slug":1027,"type":15},"QA","qa",{"name":869,"slug":870,"type":15},"2026-07-14T05:25:53.673039",{"slug":1031,"name":1031,"fn":1032,"description":1033,"org":1034,"tags":1035,"stars":952,"repoUrl":953,"updatedAt":1038},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1036,1037],{"name":843,"slug":844,"type":15},{"name":846,"slug":847,"type":15},"2026-07-14T05:25:49.362534",{"slug":1040,"name":1040,"fn":1041,"description":1042,"org":1043,"tags":1044,"stars":952,"repoUrl":953,"updatedAt":1052},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1045,1048,1049],{"name":1046,"slug":1047,"type":15},"Code Review","code-review",{"name":975,"slug":976,"type":15},{"name":1050,"slug":1051,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1054,"name":1054,"fn":1055,"description":1056,"org":1057,"tags":1058,"stars":952,"repoUrl":953,"updatedAt":1061},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1059,1060],{"name":1026,"slug":1027,"type":15},{"name":869,"slug":870,"type":15},"2026-07-14T05:25:54.928983",{"slug":1063,"name":1063,"fn":1064,"description":1065,"org":1066,"tags":1067,"stars":952,"repoUrl":953,"updatedAt":1070},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1068,1069],{"name":879,"slug":880,"type":15},{"name":962,"slug":963,"type":15},"2026-07-30T05:29:03.275638",496]