[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-sop-cr-finetuning":3,"mdc--quuh1s-key":34,"related-repo-nvidia-sop-cr-finetuning":3041,"related-org-nvidia-sop-cr-finetuning":3135},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"sop-cr-finetuning","fine-tune VLM models for SOP monitoring","Fine-tune Cosmos-Reason2 (CR2) VLM for SOP monitoring. Use when you need to launch and monitor a VLM training run with a given dataset ID.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Monitoring","monitoring","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"AI Infrastructure","ai-infrastructure",39,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fsop-monitoring-blueprints","2026-07-14T05:36:22.874218","CC-BY-4.0 AND Apache-2.0",12,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Industrial SOP Monitoring Blueprints for Training & Inference","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fsop-monitoring-blueprints\u002Ftree\u002FHEAD\u002Fagentic\u002Fsop-agentic-ft\u002Fplugins\u002Fsop-cr-finetuning-plugin\u002Fskills\u002Fsop-cr-finetuning","---\nname: sop-cr-finetuning\ndescription: Fine-tune Cosmos-Reason2 (CR2) VLM for SOP monitoring. Use when you need to launch and monitor a VLM training run with a given dataset ID.\nargument-hint: \u003Cqa_augmented_dataset_id>\nlicense: \"CC-BY-4.0 AND Apache-2.0\"\n---\n\n# SOP CR VLM Fine-tuning\n\nYou are performing Cosmos-Reason2 (CR2) VLM fine-tuning for an SOP (Standard Operating Procedure) monitoring system. The training pipeline fine-tunes a Qwen3VL-based vision-language model on augmented SOP video QA data so the model can classify operator actions from video segments.\n\nYour job is to launch training and monitor it to completion. Training logs are persisted by the training service at `results\u002F\u003Cjob_id>\u002Flog.txt`.\n\n## Input\n\nThe user provides a `qa_augmented_dataset_id` as `$ARGUMENTS`. This is the identifier for the augmented dataset to train on.\n\nIf no argument is provided, ask the user for the qa_augmented_dataset_id.\n\nFor reference on API endpoints, anomaly detection, and monitoring commands, see `${CLAUDE_SKILL_DIR}\u002Freference.md`.\n\n## Defaults\n\n- **BASE_URL:** `http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1`\n- **POLL_INTERVAL:** 60 seconds\n- **RESULTS_ROOT:** `.\u002Fassets\u002Fresults`\n\n## Fine-tuning Procedure\n\n### Phase 1: Pre-flight Checks\n\nBefore starting any training, verify readiness:\n\n1. **Check service health**:\n ```bash\n curl -s http:\u002F\u002Flocalhost:32080\u002Fhealth\n ```\n If the service is healthy, skip to step 2. If unreachable, start the container:\n ```bash\n docker compose up -d cosmos-reason-microservice\n ```\n Then poll `\u002Fhealth` with retries until ready. If the service fails to become healthy after retries, report the error and stop.\n\n2. **Check GPU availability**:\n ```bash\n nvidia-smi --query-gpu=index,name,memory.total,memory.free --format=csv,noheader\n ```\n Warn if free memory is below 60GB per GPU.\n\n3. **Check for running jobs**:\n ```bash\n curl -s http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fall_jobs\n ```\n If a job is already running, warn the user and ask whether to wait or cancel it.\n\nReport all pre-flight results before proceeding.\n\n### Phase 2: Training and Monitoring\n\n1. **Start training**:\n ```bash\n curl -s -X POST \"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstart?dataset_id=\u003Cqa_augmented_dataset_id>\" -H \"Content-Type: application\u002Fjson\"\n ```\n Use the `qa_augmented_dataset_id` from `$ARGUMENTS`. Record the `job_id` from the response.\n\n2. **Monitor training progress** by polling the status endpoint every 60 seconds:\n ```bash\n curl -s \"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstatus\u002F\u003Cjob_id>\"\n ```\n\n On each poll, extract and report:\n - `status` — queued \u002F running \u002F completed \u002F failed \u002F cancelled\n - `progress` — percentage complete\n - `current_step` \u002F `total_steps`\n - `loss` — current training loss\n\n3. **Monitor for anomalies** by checking container logs periodically:\n ```bash\n # Check for OOM\n docker compose logs --since 5m cosmos-reason-microservice 2>&1 | grep -i \"out of memory\\|OOM\\|CUDA error\"\n\n # Check for NaN\n docker compose logs --since 5m cosmos-reason-microservice 2>&1 | grep -iE \"loss.*nan|nan.*loss\" | head -5\n\n # Check GPU status\n docker compose exec cosmos-reason-microservice nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader\n ```\n\n4. **Report progress** every 10% (10%, 20%, …, 100%). Include:\n - Current loss and trend (decreasing \u002F plateau \u002F spiking)\n - Steps per second\n - Any warnings or anomalies detected (OOM, NaN, loss spikes, GPU temperature)\n\n5. **Wait for completion**. When status becomes `completed`, `failed`, or `cancelled`, proceed to Phase 3.\n\n### Phase 3: Summary\n\nAfter training completes (regardless of outcome), collect metrics, write a training report to `.\u002Fassets\u002Fresults\u002F\u003Cjob_id>\u002Ftraining_report.md`, and report to the user.\n\n#### 3a: Collect Final Metrics\n\n1. **Gather metrics** from the final status response:\n - Final loss, best loss (lowest seen during training)\n - Total steps completed, duration\n - Final status (completed \u002F failed \u002F cancelled)\n\n2. **Collect anomaly summary** from monitoring:\n - Any OOM events and recovery actions\n - Loss spikes or NaN occurrences\n - GPU memory peak usage\n\n#### 3b: Merge LoRA Adapter (if present)\n\nIf the training run used LoRA, the checkpoint is a small adapter (delta weights) that cannot be used for inference on its own and must be merged with the base model. Full fine-tuning runs skip this step — the checkpoint is already a standalone full-weight model.\n\nAfter this phase, the inference-ready model **always** lives at the same path: `assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F`. For LoRA runs, the merged full-weight model replaces the adapter at that path; the original adapter is preserved as a sibling `step_\u003CN>_lora_adapter\u002F`. Downstream evaluation reads `step_\u003CN>\u002F` via the standard `training_job_id` + `checkpoint_step` resolution and does not need to know whether the run was LoRA or full.\n\n**Path layout.** Cosmos-RL writes checkpoints to:\n```\nassets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F\n```\n`\u003Ctimestamp>` is a UTC datetime subdir (one per training run); `\u003CN>` is the global step. Pick the final step (largest `\u003CN>`) for inference.\n\n>IMPORTANT: **Run all merge commands inside the same `cosmos-reason-microservice` container that produced the adapter.**\n\n- The base model is already on disk at `\u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fweights\u002F\u003Cbase_model_dir>\u002F`.\n- The adapter is already on disk at `\u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fresults\u002F\u003Cjob_id>\u002F...`.\n\n\nContainer path prefix mapping (host ↔ container) is a fixed swap:\n```\nhost: \u003Cbp_root>\u002Fassets\u002F...\ncontainer: \u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002F...\n```\n\n**Detect LoRA vs full fine-tune.** Resolve the final-step checkpoint dir, then probe for `adapter_config.json`:\n\n```bash\nCR_CONT=$(docker ps --format '{{.Names}}' | grep cosmos-reason-microservice | head -1)\nJOB_ID=\u003Cjob_id>\nCKPT_DIR=$(docker exec $CR_CONT bash -c \\\n \"ls -d \u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fresults\u002F$JOB_ID\u002F*\u002Fsafetensors\u002Fstep_* 2>\u002Fdev\u002Fnull | sort -V | tail -1\")\necho \"checkpoint dir: $CKPT_DIR\"\n\nif docker exec $CR_CONT test -f \"$CKPT_DIR\u002Fadapter_config.json\"; then\n echo \"LoRA adapter detected — run steps 3b.1 → 3b.3.\"\nelse\n echo \"Full fine-tune detected — skip 3b.1 → 3b.3. step_\u003CN>\u002F is already an inference-ready model.\"\nfi\n```\n\nIf full fine-tune: skip the rest of this section.\n\n##### 3b.1 — Normalise `adapter_config.json` for forward-compat\n\nCosmos-RL's training stack may serialise LoRA config fields the inference-side peft doesn't accept (e.g. `r_pattern`), or write `null` where a dict is expected (e.g. `alpha_pattern: null`). Normalise defensively before loading:\n\n```bash\ndocker exec $CR_CONT python3 -c \"\nimport json\np='$CKPT_DIR\u002Fadapter_config.json'\nd=json.load(open(p))\nd.pop('r_pattern', None) # drop unknown fields\nfor k in ('alpha_pattern','rank_pattern'):\n if d.get(k) is None: d[k] = {} # null -> {} (peft expects dict)\nfor k,v in dict(bias='none', task_type='CAUSAL_LM', inference_mode=True,\n fan_in_fan_out=False, layers_to_transform=None,\n layers_pattern=None, megatron_config=None,\n megatron_core='megatron.core', loftq_config={},\n use_dora=False, init_lora_weights=True).items():\n d.setdefault(k, v) # backfill defaults\njson.dump(d, open(p,'w'), indent=2)\nprint('adapter_config.json normalised')\n\"\n```\n\n##### 3b.2 — Merge adaptor with base model\n\nWrite to a sibling `step_\u003CN>_merged_tmp\u002F` so the final atomic swap (3b.3) is crash-safe:\n\nThe base model path and the base model class are both **derived from the adapter** at run-time:\n- **Base path** comes from `adapter_config.json[\"base_model_name_or_path\"]`, which peft writes at training time. It is typically a path relative to the training process's working directory (`\u002Fworkspace\u002Fsop-cr-ftms`), e.g. `.\u002Fassets\u002Fweights\u002FCosmos-Reason2-2B`. Resolve it against that working dir if it isn't already absolute.\n- **Model class** comes from `AutoModelForImageTextToText`, which reads `architectures` from the base model's `config.json` and instantiates the right class (e.g. `Qwen3VLForConditionalGeneration` for Cosmos-Reason2-2B, `Qwen2_5_VLForConditionalGeneration` for older variants).\n\n```bash\ndocker exec -w \u002Fworkspace\u002Fsop-cr-ftms $CR_CONT python3 -c \"\nimport json, os, torch\nfrom transformers import AutoModelForImageTextToText, AutoProcessor\nfrom peft import PeftModel\n\nadapter_path = '$CKPT_DIR'\ntmp_path = adapter_path + '_merged_tmp'\n\n# Resolve the base model path from the adapter's own metadata.\nwith open(os.path.join(adapter_path, 'adapter_config.json')) as f:\n base_rel = json.load(f)['base_model_name_or_path']\nbase_path = (base_rel if os.path.isabs(base_rel)\n else os.path.normpath(os.path.join('\u002Fworkspace\u002Fsop-cr-ftms', base_rel)))\nprint(f'base model: {base_path}', flush=True)\n\nprint('loading base...', flush=True)\nbase = AutoModelForImageTextToText.from_pretrained(\n base_path, dtype=torch.bfloat16, device_map='cpu')\nprint('loading adapter...', flush=True)\nmodel = PeftModel.from_pretrained(base, adapter_path)\nprint('merging...', flush=True)\nmerged = model.merge_and_unload()\n\nos.makedirs(tmp_path, exist_ok=True)\nmerged.save_pretrained(tmp_path, safe_serialization=True)\n# Processor\u002Fpreprocessor configs live with the base model, not the adapter.\n# The adapter dir only carries tokenizer files.\nAutoProcessor.from_pretrained(base_path).save_pretrained(tmp_path)\nprint('merged ->', tmp_path)\n\"\n```\n\nNotes on the kwargs:\n- `device_map='cpu'` — merge is memory-bandwidth-bound and uses ~6 GB CPU RAM. Keeps GPUs free for concurrent training\u002Feval.\n- `dtype=torch.bfloat16` — matches the precision the LoRA was trained against.\n- The processor (`AutoProcessor`) is loaded from the **base** model path, not the adapter path. Vision\u002Fvideo preprocessor configs live with the base; the adapter dir only carries tokenizer + adapter files.\n\n##### 3b.3 — Atomic swap into `step_\u003CN>\u002F`\n\nEvaluation-ms resolves checkpoints by `safetensors\u002Fstep_\u003CN>\u002F` and calls `from_pretrained` on that exact path; it has no knowledge of `merged_model\u002F` or other subdirs. Swap the merged result in and keep the original adapter as a sibling:\n\n```bash\ndocker exec $CR_CONT bash -c \"\n STEP_DIR='$CKPT_DIR'\n PARENT=\\$(dirname \\$STEP_DIR)\n STEP=\\$(basename \\$STEP_DIR)\n mv \\$STEP_DIR \\${PARENT}\u002F\\${STEP}_lora_adapter\n mv \\${PARENT}\u002F\\${STEP}_merged_tmp \\$STEP_DIR\n\"\n```\n\n##### 3b.4 — Verify\n\nListing the new `step_\u003CN>\u002F` should show full-model files, not adapter files:\n\n```bash\ndocker exec $CR_CONT ls \"$CKPT_DIR\"\n```\nExpect `config.json`, `model.safetensors` (or sharded `model-00001-of-NNNNN.safetensors`), `tokenizer.json`, `preprocessor_config.json`, `video_preprocessor_config.json`, generation\u002Fspecial-token configs. The presence of `adapter_model.safetensors` here means the merge did not land — recheck 3b.3.\n\n\n#### 3c: Write Training Report\n\nSave a formatted training report to `.\u002Fassets\u002Fresults\u002F\u003Cjob_id>\u002Ftraining_report.md`:\n\n```markdown\n# Training Report\n\n## Overview\n| Field | Value |\n|-------|-------|\n| Dataset | \u003Cqa_augmented_dataset_id> |\n| Job ID | \u003Cjob_id> |\n| Status | completed \u002F failed \u002F cancelled |\n| Duration | Xh Xm |\n\n## Training Metrics\n| Metric | Value |\n|--------|-------|\n| Final Loss | X.XX |\n| Best Loss | X.XX (step N) |\n| Total Steps | N \u002F N |\n| Steps\u002Fsec | X.XX |\n| GPU Memory Peak | XX% |\n\n## Anomalies\n- List any OOM events, loss spikes, NaN occurrences, or GPU warnings detected during training.\n- \"None\" if no anomalies were detected.\n\n## Artifacts\n- **Inference-ready checkpoint:** `assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F` — always, regardless of training mode (Phase 3b merges LoRA into the base in place; full fine-tune writes here directly).\n- **LoRA adapter backup (LoRA runs only):** `assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>_lora_adapter\u002F` — the unmerged adapter, kept for reproducibility\u002Faudit.\n- **Training log:** `results\u002F\u003Cjob_id>\u002Flog.txt`\n```\n\n#### 3d: Output Summary\n\nPrint the training report content to the user as well. If training failed, include the error details.\n",{"data":35,"body":37},{"name":4,"description":6,"argument-hint":36,"license":26},"\u003Cqa_augmented_dataset_id>",{"type":38,"children":39},"root",[40,49,55,69,76,97,102,114,120,167,173,180,185,363,368,374,871,877,890,897,956,962,967,1019,1029,1039,1065,1087,1114,1119,1128,1145,1470,1475,1489,1518,1686,1692,1705,1717,1803,2097,2102,2147,2158,2187,2364,2370,2382,2421,2478,2484,2495,3024,3030,3035],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"sop-cr-vlm-fine-tuning",[46],{"type":47,"value":48},"text","SOP CR VLM Fine-tuning",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"You are performing Cosmos-Reason2 (CR2) VLM fine-tuning for an SOP (Standard Operating Procedure) monitoring system. The training pipeline fine-tunes a Qwen3VL-based vision-language model on augmented SOP video QA data so the model can classify operator actions from video segments.",{"type":41,"tag":50,"props":56,"children":57},{},[58,60,67],{"type":47,"value":59},"Your job is to launch training and monitor it to completion. Training logs are persisted by the training service at ",{"type":41,"tag":61,"props":62,"children":64},"code",{"className":63},[],[65],{"type":47,"value":66},"results\u002F\u003Cjob_id>\u002Flog.txt",{"type":47,"value":68},".",{"type":41,"tag":70,"props":71,"children":73},"h2",{"id":72},"input",[74],{"type":47,"value":75},"Input",{"type":41,"tag":50,"props":77,"children":78},{},[79,81,87,89,95],{"type":47,"value":80},"The user provides a ",{"type":41,"tag":61,"props":82,"children":84},{"className":83},[],[85],{"type":47,"value":86},"qa_augmented_dataset_id",{"type":47,"value":88}," as ",{"type":41,"tag":61,"props":90,"children":92},{"className":91},[],[93],{"type":47,"value":94},"$ARGUMENTS",{"type":47,"value":96},". This is the identifier for the augmented dataset to train on.",{"type":41,"tag":50,"props":98,"children":99},{},[100],{"type":47,"value":101},"If no argument is provided, ask the user for the qa_augmented_dataset_id.",{"type":41,"tag":50,"props":103,"children":104},{},[105,107,113],{"type":47,"value":106},"For reference on API endpoints, anomaly detection, and monitoring commands, see ",{"type":41,"tag":61,"props":108,"children":110},{"className":109},[],[111],{"type":47,"value":112},"${CLAUDE_SKILL_DIR}\u002Freference.md",{"type":47,"value":68},{"type":41,"tag":70,"props":115,"children":117},{"id":116},"defaults",[118],{"type":47,"value":119},"Defaults",{"type":41,"tag":121,"props":122,"children":123},"ul",{},[124,142,152],{"type":41,"tag":125,"props":126,"children":127},"li",{},[128,134,136],{"type":41,"tag":129,"props":130,"children":131},"strong",{},[132],{"type":47,"value":133},"BASE_URL:",{"type":47,"value":135}," ",{"type":41,"tag":61,"props":137,"children":139},{"className":138},[],[140],{"type":47,"value":141},"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1",{"type":41,"tag":125,"props":143,"children":144},{},[145,150],{"type":41,"tag":129,"props":146,"children":147},{},[148],{"type":47,"value":149},"POLL_INTERVAL:",{"type":47,"value":151}," 60 seconds",{"type":41,"tag":125,"props":153,"children":154},{},[155,160,161],{"type":41,"tag":129,"props":156,"children":157},{},[158],{"type":47,"value":159},"RESULTS_ROOT:",{"type":47,"value":135},{"type":41,"tag":61,"props":162,"children":164},{"className":163},[],[165],{"type":47,"value":166},".\u002Fassets\u002Fresults",{"type":41,"tag":70,"props":168,"children":170},{"id":169},"fine-tuning-procedure",[171],{"type":47,"value":172},"Fine-tuning Procedure",{"type":41,"tag":174,"props":175,"children":177},"h3",{"id":176},"phase-1-pre-flight-checks",[178],{"type":47,"value":179},"Phase 1: Pre-flight Checks",{"type":41,"tag":50,"props":181,"children":182},{},[183],{"type":47,"value":184},"Before starting any training, verify readiness:",{"type":41,"tag":186,"props":187,"children":188},"ol",{},[189,287,326],{"type":41,"tag":125,"props":190,"children":191},{},[192,197,199,233,237,239,274,277,279,285],{"type":41,"tag":129,"props":193,"children":194},{},[195],{"type":47,"value":196},"Check service health",{"type":47,"value":198},":",{"type":41,"tag":200,"props":201,"children":206},"pre",{"className":202,"code":203,"language":204,"meta":205,"style":205},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -s http:\u002F\u002Flocalhost:32080\u002Fhealth\n","bash","",[207],{"type":41,"tag":61,"props":208,"children":209},{"__ignoreMap":205},[210],{"type":41,"tag":211,"props":212,"children":215},"span",{"class":213,"line":214},"line",1,[216,222,228],{"type":41,"tag":211,"props":217,"children":219},{"style":218},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[220],{"type":47,"value":221},"curl",{"type":41,"tag":211,"props":223,"children":225},{"style":224},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[226],{"type":47,"value":227}," -s",{"type":41,"tag":211,"props":229,"children":230},{"style":224},[231],{"type":47,"value":232}," http:\u002F\u002Flocalhost:32080\u002Fhealth\n",{"type":41,"tag":234,"props":235,"children":236},"br",{},[],{"type":47,"value":238},"If the service is healthy, skip to step 2. If unreachable, start the container:",{"type":41,"tag":200,"props":240,"children":242},{"className":202,"code":241,"language":204,"meta":205,"style":205},"docker compose up -d cosmos-reason-microservice\n",[243],{"type":41,"tag":61,"props":244,"children":245},{"__ignoreMap":205},[246],{"type":41,"tag":211,"props":247,"children":248},{"class":213,"line":214},[249,254,259,264,269],{"type":41,"tag":211,"props":250,"children":251},{"style":218},[252],{"type":47,"value":253},"docker",{"type":41,"tag":211,"props":255,"children":256},{"style":224},[257],{"type":47,"value":258}," compose",{"type":41,"tag":211,"props":260,"children":261},{"style":224},[262],{"type":47,"value":263}," up",{"type":41,"tag":211,"props":265,"children":266},{"style":224},[267],{"type":47,"value":268}," -d",{"type":41,"tag":211,"props":270,"children":271},{"style":224},[272],{"type":47,"value":273}," cosmos-reason-microservice\n",{"type":41,"tag":234,"props":275,"children":276},{},[],{"type":47,"value":278},"Then poll ",{"type":41,"tag":61,"props":280,"children":282},{"className":281},[],[283],{"type":47,"value":284},"\u002Fhealth",{"type":47,"value":286}," with retries until ready. If the service fails to become healthy after retries, report the error and stop.",{"type":41,"tag":125,"props":288,"children":289},{},[290,295,296,321,324],{"type":41,"tag":129,"props":291,"children":292},{},[293],{"type":47,"value":294},"Check GPU availability",{"type":47,"value":198},{"type":41,"tag":200,"props":297,"children":299},{"className":202,"code":298,"language":204,"meta":205,"style":205},"nvidia-smi --query-gpu=index,name,memory.total,memory.free --format=csv,noheader\n",[300],{"type":41,"tag":61,"props":301,"children":302},{"__ignoreMap":205},[303],{"type":41,"tag":211,"props":304,"children":305},{"class":213,"line":214},[306,311,316],{"type":41,"tag":211,"props":307,"children":308},{"style":218},[309],{"type":47,"value":310},"nvidia-smi",{"type":41,"tag":211,"props":312,"children":313},{"style":224},[314],{"type":47,"value":315}," --query-gpu=index,name,memory.total,memory.free",{"type":41,"tag":211,"props":317,"children":318},{"style":224},[319],{"type":47,"value":320}," --format=csv,noheader\n",{"type":41,"tag":234,"props":322,"children":323},{},[],{"type":47,"value":325},"Warn if free memory is below 60GB per GPU.",{"type":41,"tag":125,"props":327,"children":328},{},[329,334,335,358,361],{"type":41,"tag":129,"props":330,"children":331},{},[332],{"type":47,"value":333},"Check for running jobs",{"type":47,"value":198},{"type":41,"tag":200,"props":336,"children":338},{"className":202,"code":337,"language":204,"meta":205,"style":205},"curl -s http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fall_jobs\n",[339],{"type":41,"tag":61,"props":340,"children":341},{"__ignoreMap":205},[342],{"type":41,"tag":211,"props":343,"children":344},{"class":213,"line":214},[345,349,353],{"type":41,"tag":211,"props":346,"children":347},{"style":218},[348],{"type":47,"value":221},{"type":41,"tag":211,"props":350,"children":351},{"style":224},[352],{"type":47,"value":227},{"type":41,"tag":211,"props":354,"children":355},{"style":224},[356],{"type":47,"value":357}," http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fall_jobs\n",{"type":41,"tag":234,"props":359,"children":360},{},[],{"type":47,"value":362},"If a job is already running, warn the user and ask whether to wait or cancel it.",{"type":41,"tag":50,"props":364,"children":365},{},[366],{"type":47,"value":367},"Report all pre-flight results before proceeding.",{"type":41,"tag":174,"props":369,"children":371},{"id":370},"phase-2-training-and-monitoring",[372],{"type":47,"value":373},"Phase 2: Training and Monitoring",{"type":41,"tag":186,"props":375,"children":376},{},[377,476,575,809,837],{"type":41,"tag":125,"props":378,"children":379},{},[380,385,386,449,452,454,459,461,466,468,474],{"type":41,"tag":129,"props":381,"children":382},{},[383],{"type":47,"value":384},"Start training",{"type":47,"value":198},{"type":41,"tag":200,"props":387,"children":389},{"className":202,"code":388,"language":204,"meta":205,"style":205},"curl -s -X POST \"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstart?dataset_id=\u003Cqa_augmented_dataset_id>\" -H \"Content-Type: application\u002Fjson\"\n",[390],{"type":41,"tag":61,"props":391,"children":392},{"__ignoreMap":205},[393],{"type":41,"tag":211,"props":394,"children":395},{"class":213,"line":214},[396,400,404,409,414,420,425,430,435,439,444],{"type":41,"tag":211,"props":397,"children":398},{"style":218},[399],{"type":47,"value":221},{"type":41,"tag":211,"props":401,"children":402},{"style":224},[403],{"type":47,"value":227},{"type":41,"tag":211,"props":405,"children":406},{"style":224},[407],{"type":47,"value":408}," -X",{"type":41,"tag":211,"props":410,"children":411},{"style":224},[412],{"type":47,"value":413}," POST",{"type":41,"tag":211,"props":415,"children":417},{"style":416},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[418],{"type":47,"value":419}," \"",{"type":41,"tag":211,"props":421,"children":422},{"style":224},[423],{"type":47,"value":424},"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstart?dataset_id=\u003Cqa_augmented_dataset_id>",{"type":41,"tag":211,"props":426,"children":427},{"style":416},[428],{"type":47,"value":429},"\"",{"type":41,"tag":211,"props":431,"children":432},{"style":224},[433],{"type":47,"value":434}," -H",{"type":41,"tag":211,"props":436,"children":437},{"style":416},[438],{"type":47,"value":419},{"type":41,"tag":211,"props":440,"children":441},{"style":224},[442],{"type":47,"value":443},"Content-Type: application\u002Fjson",{"type":41,"tag":211,"props":445,"children":446},{"style":416},[447],{"type":47,"value":448},"\"\n",{"type":41,"tag":234,"props":450,"children":451},{},[],{"type":47,"value":453},"Use the ",{"type":41,"tag":61,"props":455,"children":457},{"className":456},[],[458],{"type":47,"value":86},{"type":47,"value":460}," from ",{"type":41,"tag":61,"props":462,"children":464},{"className":463},[],[465],{"type":47,"value":94},{"type":47,"value":467},". Record the ",{"type":41,"tag":61,"props":469,"children":471},{"className":470},[],[472],{"type":47,"value":473},"job_id",{"type":47,"value":475}," from the response.",{"type":41,"tag":125,"props":477,"children":478},{},[479,484,486,517,520,522],{"type":41,"tag":129,"props":480,"children":481},{},[482],{"type":47,"value":483},"Monitor training progress",{"type":47,"value":485}," by polling the status endpoint every 60 seconds:",{"type":41,"tag":200,"props":487,"children":489},{"className":202,"code":488,"language":204,"meta":205,"style":205},"curl -s \"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstatus\u002F\u003Cjob_id>\"\n",[490],{"type":41,"tag":61,"props":491,"children":492},{"__ignoreMap":205},[493],{"type":41,"tag":211,"props":494,"children":495},{"class":213,"line":214},[496,500,504,508,513],{"type":41,"tag":211,"props":497,"children":498},{"style":218},[499],{"type":47,"value":221},{"type":41,"tag":211,"props":501,"children":502},{"style":224},[503],{"type":47,"value":227},{"type":41,"tag":211,"props":505,"children":506},{"style":416},[507],{"type":47,"value":419},{"type":41,"tag":211,"props":509,"children":510},{"style":224},[511],{"type":47,"value":512},"http:\u002F\u002Flocalhost:32080\u002Fapi\u002Fv1\u002Ffine-tuning\u002Fstatus\u002F\u003Cjob_id>",{"type":41,"tag":211,"props":514,"children":515},{"style":416},[516],{"type":47,"value":448},{"type":41,"tag":234,"props":518,"children":519},{},[],{"type":47,"value":521},"On each poll, extract and report:",{"type":41,"tag":121,"props":523,"children":524},{},[525,536,547,564],{"type":41,"tag":125,"props":526,"children":527},{},[528,534],{"type":41,"tag":61,"props":529,"children":531},{"className":530},[],[532],{"type":47,"value":533},"status",{"type":47,"value":535}," — queued \u002F running \u002F completed \u002F failed \u002F cancelled",{"type":41,"tag":125,"props":537,"children":538},{},[539,545],{"type":41,"tag":61,"props":540,"children":542},{"className":541},[],[543],{"type":47,"value":544},"progress",{"type":47,"value":546}," — percentage complete",{"type":41,"tag":125,"props":548,"children":549},{},[550,556,558],{"type":41,"tag":61,"props":551,"children":553},{"className":552},[],[554],{"type":47,"value":555},"current_step",{"type":47,"value":557}," \u002F ",{"type":41,"tag":61,"props":559,"children":561},{"className":560},[],[562],{"type":47,"value":563},"total_steps",{"type":41,"tag":125,"props":565,"children":566},{},[567,573],{"type":41,"tag":61,"props":568,"children":570},{"className":569},[],[571],{"type":47,"value":572},"loss",{"type":47,"value":574}," — current training loss",{"type":41,"tag":125,"props":576,"children":577},{},[578,583,585],{"type":41,"tag":129,"props":579,"children":580},{},[581],{"type":47,"value":582},"Monitor for anomalies",{"type":47,"value":584}," by checking container logs periodically:",{"type":41,"tag":200,"props":586,"children":588},{"className":202,"code":587,"language":204,"meta":205,"style":205},"# Check for OOM\ndocker compose logs --since 5m cosmos-reason-microservice 2>&1 | grep -i \"out of memory\\|OOM\\|CUDA error\"\n\n# Check for NaN\ndocker compose logs --since 5m cosmos-reason-microservice 2>&1 | grep -iE \"loss.*nan|nan.*loss\" | head -5\n\n# Check GPU status\ndocker compose exec cosmos-reason-microservice nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu --format=csv,noheader\n",[589],{"type":41,"tag":61,"props":590,"children":591},{"__ignoreMap":205},[592,601,666,676,685,757,765,774],{"type":41,"tag":211,"props":593,"children":594},{"class":213,"line":214},[595],{"type":41,"tag":211,"props":596,"children":598},{"style":597},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[599],{"type":47,"value":600},"# Check for OOM\n",{"type":41,"tag":211,"props":602,"children":604},{"class":213,"line":603},2,[605,609,613,618,623,628,633,638,643,648,653,657,662],{"type":41,"tag":211,"props":606,"children":607},{"style":218},[608],{"type":47,"value":253},{"type":41,"tag":211,"props":610,"children":611},{"style":224},[612],{"type":47,"value":258},{"type":41,"tag":211,"props":614,"children":615},{"style":224},[616],{"type":47,"value":617}," logs",{"type":41,"tag":211,"props":619,"children":620},{"style":224},[621],{"type":47,"value":622}," --since",{"type":41,"tag":211,"props":624,"children":625},{"style":224},[626],{"type":47,"value":627}," 5m",{"type":41,"tag":211,"props":629,"children":630},{"style":224},[631],{"type":47,"value":632}," cosmos-reason-microservice",{"type":41,"tag":211,"props":634,"children":635},{"style":416},[636],{"type":47,"value":637}," 2>&1",{"type":41,"tag":211,"props":639,"children":640},{"style":416},[641],{"type":47,"value":642}," |",{"type":41,"tag":211,"props":644,"children":645},{"style":218},[646],{"type":47,"value":647}," grep",{"type":41,"tag":211,"props":649,"children":650},{"style":224},[651],{"type":47,"value":652}," -i",{"type":41,"tag":211,"props":654,"children":655},{"style":416},[656],{"type":47,"value":419},{"type":41,"tag":211,"props":658,"children":659},{"style":224},[660],{"type":47,"value":661},"out of memory\\|OOM\\|CUDA error",{"type":41,"tag":211,"props":663,"children":664},{"style":416},[665],{"type":47,"value":448},{"type":41,"tag":211,"props":667,"children":669},{"class":213,"line":668},3,[670],{"type":41,"tag":211,"props":671,"children":673},{"emptyLinePlaceholder":672},true,[674],{"type":47,"value":675},"\n",{"type":41,"tag":211,"props":677,"children":679},{"class":213,"line":678},4,[680],{"type":41,"tag":211,"props":681,"children":682},{"style":597},[683],{"type":47,"value":684},"# Check for NaN\n",{"type":41,"tag":211,"props":686,"children":688},{"class":213,"line":687},5,[689,693,697,701,705,709,713,717,721,725,730,734,739,743,747,752],{"type":41,"tag":211,"props":690,"children":691},{"style":218},[692],{"type":47,"value":253},{"type":41,"tag":211,"props":694,"children":695},{"style":224},[696],{"type":47,"value":258},{"type":41,"tag":211,"props":698,"children":699},{"style":224},[700],{"type":47,"value":617},{"type":41,"tag":211,"props":702,"children":703},{"style":224},[704],{"type":47,"value":622},{"type":41,"tag":211,"props":706,"children":707},{"style":224},[708],{"type":47,"value":627},{"type":41,"tag":211,"props":710,"children":711},{"style":224},[712],{"type":47,"value":632},{"type":41,"tag":211,"props":714,"children":715},{"style":416},[716],{"type":47,"value":637},{"type":41,"tag":211,"props":718,"children":719},{"style":416},[720],{"type":47,"value":642},{"type":41,"tag":211,"props":722,"children":723},{"style":218},[724],{"type":47,"value":647},{"type":41,"tag":211,"props":726,"children":727},{"style":224},[728],{"type":47,"value":729}," -iE",{"type":41,"tag":211,"props":731,"children":732},{"style":416},[733],{"type":47,"value":419},{"type":41,"tag":211,"props":735,"children":736},{"style":224},[737],{"type":47,"value":738},"loss.*nan|nan.*loss",{"type":41,"tag":211,"props":740,"children":741},{"style":416},[742],{"type":47,"value":429},{"type":41,"tag":211,"props":744,"children":745},{"style":416},[746],{"type":47,"value":642},{"type":41,"tag":211,"props":748,"children":749},{"style":218},[750],{"type":47,"value":751}," head",{"type":41,"tag":211,"props":753,"children":754},{"style":224},[755],{"type":47,"value":756}," -5\n",{"type":41,"tag":211,"props":758,"children":760},{"class":213,"line":759},6,[761],{"type":41,"tag":211,"props":762,"children":763},{"emptyLinePlaceholder":672},[764],{"type":47,"value":675},{"type":41,"tag":211,"props":766,"children":768},{"class":213,"line":767},7,[769],{"type":41,"tag":211,"props":770,"children":771},{"style":597},[772],{"type":47,"value":773},"# Check GPU status\n",{"type":41,"tag":211,"props":775,"children":777},{"class":213,"line":776},8,[778,782,786,791,795,800,805],{"type":41,"tag":211,"props":779,"children":780},{"style":218},[781],{"type":47,"value":253},{"type":41,"tag":211,"props":783,"children":784},{"style":224},[785],{"type":47,"value":258},{"type":41,"tag":211,"props":787,"children":788},{"style":224},[789],{"type":47,"value":790}," exec",{"type":41,"tag":211,"props":792,"children":793},{"style":224},[794],{"type":47,"value":632},{"type":41,"tag":211,"props":796,"children":797},{"style":224},[798],{"type":47,"value":799}," nvidia-smi",{"type":41,"tag":211,"props":801,"children":802},{"style":224},[803],{"type":47,"value":804}," --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu",{"type":41,"tag":211,"props":806,"children":807},{"style":224},[808],{"type":47,"value":320},{"type":41,"tag":125,"props":810,"children":811},{},[812,817,819],{"type":41,"tag":129,"props":813,"children":814},{},[815],{"type":47,"value":816},"Report progress",{"type":47,"value":818}," every 10% (10%, 20%, …, 100%). Include:",{"type":41,"tag":121,"props":820,"children":821},{},[822,827,832],{"type":41,"tag":125,"props":823,"children":824},{},[825],{"type":47,"value":826},"Current loss and trend (decreasing \u002F plateau \u002F spiking)",{"type":41,"tag":125,"props":828,"children":829},{},[830],{"type":47,"value":831},"Steps per second",{"type":41,"tag":125,"props":833,"children":834},{},[835],{"type":47,"value":836},"Any warnings or anomalies detected (OOM, NaN, loss spikes, GPU temperature)",{"type":41,"tag":125,"props":838,"children":839},{},[840,845,847,853,855,861,863,869],{"type":41,"tag":129,"props":841,"children":842},{},[843],{"type":47,"value":844},"Wait for completion",{"type":47,"value":846},". When status becomes ",{"type":41,"tag":61,"props":848,"children":850},{"className":849},[],[851],{"type":47,"value":852},"completed",{"type":47,"value":854},", ",{"type":41,"tag":61,"props":856,"children":858},{"className":857},[],[859],{"type":47,"value":860},"failed",{"type":47,"value":862},", or ",{"type":41,"tag":61,"props":864,"children":866},{"className":865},[],[867],{"type":47,"value":868},"cancelled",{"type":47,"value":870},", proceed to Phase 3.",{"type":41,"tag":174,"props":872,"children":874},{"id":873},"phase-3-summary",[875],{"type":47,"value":876},"Phase 3: Summary",{"type":41,"tag":50,"props":878,"children":879},{},[880,882,888],{"type":47,"value":881},"After training completes (regardless of outcome), collect metrics, write a training report to ",{"type":41,"tag":61,"props":883,"children":885},{"className":884},[],[886],{"type":47,"value":887},".\u002Fassets\u002Fresults\u002F\u003Cjob_id>\u002Ftraining_report.md",{"type":47,"value":889},", and report to the user.",{"type":41,"tag":891,"props":892,"children":894},"h4",{"id":893},"_3a-collect-final-metrics",[895],{"type":47,"value":896},"3a: Collect Final Metrics",{"type":41,"tag":186,"props":898,"children":899},{},[900,928],{"type":41,"tag":125,"props":901,"children":902},{},[903,908,910],{"type":41,"tag":129,"props":904,"children":905},{},[906],{"type":47,"value":907},"Gather metrics",{"type":47,"value":909}," from the final status response:",{"type":41,"tag":121,"props":911,"children":912},{},[913,918,923],{"type":41,"tag":125,"props":914,"children":915},{},[916],{"type":47,"value":917},"Final loss, best loss (lowest seen during training)",{"type":41,"tag":125,"props":919,"children":920},{},[921],{"type":47,"value":922},"Total steps completed, duration",{"type":41,"tag":125,"props":924,"children":925},{},[926],{"type":47,"value":927},"Final status (completed \u002F failed \u002F cancelled)",{"type":41,"tag":125,"props":929,"children":930},{},[931,936,938],{"type":41,"tag":129,"props":932,"children":933},{},[934],{"type":47,"value":935},"Collect anomaly summary",{"type":47,"value":937}," from monitoring:",{"type":41,"tag":121,"props":939,"children":940},{},[941,946,951],{"type":41,"tag":125,"props":942,"children":943},{},[944],{"type":47,"value":945},"Any OOM events and recovery actions",{"type":41,"tag":125,"props":947,"children":948},{},[949],{"type":47,"value":950},"Loss spikes or NaN occurrences",{"type":41,"tag":125,"props":952,"children":953},{},[954],{"type":47,"value":955},"GPU memory peak usage",{"type":41,"tag":891,"props":957,"children":959},{"id":958},"_3b-merge-lora-adapter-if-present",[960],{"type":47,"value":961},"3b: Merge LoRA Adapter (if present)",{"type":41,"tag":50,"props":963,"children":964},{},[965],{"type":47,"value":966},"If the training run used LoRA, the checkpoint is a small adapter (delta weights) that cannot be used for inference on its own and must be merged with the base model. Full fine-tuning runs skip this step — the checkpoint is already a standalone full-weight model.",{"type":41,"tag":50,"props":968,"children":969},{},[970,972,977,979,985,987,993,995,1001,1003,1009,1011,1017],{"type":47,"value":971},"After this phase, the inference-ready model ",{"type":41,"tag":129,"props":973,"children":974},{},[975],{"type":47,"value":976},"always",{"type":47,"value":978}," lives at the same path: ",{"type":41,"tag":61,"props":980,"children":982},{"className":981},[],[983],{"type":47,"value":984},"assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F",{"type":47,"value":986},". For LoRA runs, the merged full-weight model replaces the adapter at that path; the original adapter is preserved as a sibling ",{"type":41,"tag":61,"props":988,"children":990},{"className":989},[],[991],{"type":47,"value":992},"step_\u003CN>_lora_adapter\u002F",{"type":47,"value":994},". Downstream evaluation reads ",{"type":41,"tag":61,"props":996,"children":998},{"className":997},[],[999],{"type":47,"value":1000},"step_\u003CN>\u002F",{"type":47,"value":1002}," via the standard ",{"type":41,"tag":61,"props":1004,"children":1006},{"className":1005},[],[1007],{"type":47,"value":1008},"training_job_id",{"type":47,"value":1010}," + ",{"type":41,"tag":61,"props":1012,"children":1014},{"className":1013},[],[1015],{"type":47,"value":1016},"checkpoint_step",{"type":47,"value":1018}," resolution and does not need to know whether the run was LoRA or full.",{"type":41,"tag":50,"props":1020,"children":1021},{},[1022,1027],{"type":41,"tag":129,"props":1023,"children":1024},{},[1025],{"type":47,"value":1026},"Path layout.",{"type":47,"value":1028}," Cosmos-RL writes checkpoints to:",{"type":41,"tag":200,"props":1030,"children":1034},{"className":1031,"code":1033,"language":47},[1032],"language-text","assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F\n",[1035],{"type":41,"tag":61,"props":1036,"children":1037},{"__ignoreMap":205},[1038],{"type":47,"value":1033},{"type":41,"tag":50,"props":1040,"children":1041},{},[1042,1048,1050,1056,1058,1063],{"type":41,"tag":61,"props":1043,"children":1045},{"className":1044},[],[1046],{"type":47,"value":1047},"\u003Ctimestamp>",{"type":47,"value":1049}," is a UTC datetime subdir (one per training run); ",{"type":41,"tag":61,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":47,"value":1055},"\u003CN>",{"type":47,"value":1057}," is the global step. Pick the final step (largest ",{"type":41,"tag":61,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":47,"value":1055},{"type":47,"value":1064},") for inference.",{"type":41,"tag":1066,"props":1067,"children":1068},"blockquote",{},[1069],{"type":41,"tag":50,"props":1070,"children":1071},{},[1072,1074],{"type":47,"value":1073},"IMPORTANT: ",{"type":41,"tag":129,"props":1075,"children":1076},{},[1077,1079,1085],{"type":47,"value":1078},"Run all merge commands inside the same ",{"type":41,"tag":61,"props":1080,"children":1082},{"className":1081},[],[1083],{"type":47,"value":1084},"cosmos-reason-microservice",{"type":47,"value":1086}," container that produced the adapter.",{"type":41,"tag":121,"props":1088,"children":1089},{},[1090,1102],{"type":41,"tag":125,"props":1091,"children":1092},{},[1093,1095,1101],{"type":47,"value":1094},"The base model is already on disk at ",{"type":41,"tag":61,"props":1096,"children":1098},{"className":1097},[],[1099],{"type":47,"value":1100},"\u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fweights\u002F\u003Cbase_model_dir>\u002F",{"type":47,"value":68},{"type":41,"tag":125,"props":1103,"children":1104},{},[1105,1107,1113],{"type":47,"value":1106},"The adapter is already on disk at ",{"type":41,"tag":61,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":47,"value":1112},"\u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fresults\u002F\u003Cjob_id>\u002F...",{"type":47,"value":68},{"type":41,"tag":50,"props":1115,"children":1116},{},[1117],{"type":47,"value":1118},"Container path prefix mapping (host ↔ container) is a fixed swap:",{"type":41,"tag":200,"props":1120,"children":1123},{"className":1121,"code":1122,"language":47},[1032],"host: \u003Cbp_root>\u002Fassets\u002F...\ncontainer: \u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002F...\n",[1124],{"type":41,"tag":61,"props":1125,"children":1126},{"__ignoreMap":205},[1127],{"type":47,"value":1122},{"type":41,"tag":50,"props":1129,"children":1130},{},[1131,1136,1138,1144],{"type":41,"tag":129,"props":1132,"children":1133},{},[1134],{"type":47,"value":1135},"Detect LoRA vs full fine-tune.",{"type":47,"value":1137}," Resolve the final-step checkpoint dir, then probe for ",{"type":41,"tag":61,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":47,"value":1143},"adapter_config.json",{"type":47,"value":198},{"type":41,"tag":200,"props":1146,"children":1148},{"className":202,"code":1147,"language":204,"meta":205,"style":205},"CR_CONT=$(docker ps --format '{{.Names}}' | grep cosmos-reason-microservice | head -1)\nJOB_ID=\u003Cjob_id>\nCKPT_DIR=$(docker exec $CR_CONT bash -c \\\n \"ls -d \u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fresults\u002F$JOB_ID\u002F*\u002Fsafetensors\u002Fstep_* 2>\u002Fdev\u002Fnull | sort -V | tail -1\")\necho \"checkpoint dir: $CKPT_DIR\"\n\nif docker exec $CR_CONT test -f \"$CKPT_DIR\u002Fadapter_config.json\"; then\n echo \"LoRA adapter detected — run steps 3b.1 → 3b.3.\"\nelse\n echo \"Full fine-tune detected — skip 3b.1 → 3b.3. step_\u003CN>\u002F is already an inference-ready model.\"\nfi\n",[1149],{"type":41,"tag":61,"props":1150,"children":1151},{"__ignoreMap":205},[1152,1225,1247,1286,1317,1344,1351,1410,1431,1440,1461],{"type":41,"tag":211,"props":1153,"children":1154},{"class":213,"line":214},[1155,1161,1166,1170,1175,1180,1185,1190,1195,1199,1203,1207,1211,1215,1220],{"type":41,"tag":211,"props":1156,"children":1158},{"style":1157},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1159],{"type":47,"value":1160},"CR_CONT",{"type":41,"tag":211,"props":1162,"children":1163},{"style":416},[1164],{"type":47,"value":1165},"=$(",{"type":41,"tag":211,"props":1167,"children":1168},{"style":218},[1169],{"type":47,"value":253},{"type":41,"tag":211,"props":1171,"children":1172},{"style":224},[1173],{"type":47,"value":1174}," ps",{"type":41,"tag":211,"props":1176,"children":1177},{"style":224},[1178],{"type":47,"value":1179}," --format",{"type":41,"tag":211,"props":1181,"children":1182},{"style":416},[1183],{"type":47,"value":1184}," '",{"type":41,"tag":211,"props":1186,"children":1187},{"style":224},[1188],{"type":47,"value":1189},"{{.Names}}",{"type":41,"tag":211,"props":1191,"children":1192},{"style":416},[1193],{"type":47,"value":1194},"'",{"type":41,"tag":211,"props":1196,"children":1197},{"style":416},[1198],{"type":47,"value":642},{"type":41,"tag":211,"props":1200,"children":1201},{"style":218},[1202],{"type":47,"value":647},{"type":41,"tag":211,"props":1204,"children":1205},{"style":224},[1206],{"type":47,"value":632},{"type":41,"tag":211,"props":1208,"children":1209},{"style":416},[1210],{"type":47,"value":642},{"type":41,"tag":211,"props":1212,"children":1213},{"style":218},[1214],{"type":47,"value":751},{"type":41,"tag":211,"props":1216,"children":1217},{"style":224},[1218],{"type":47,"value":1219}," -1",{"type":41,"tag":211,"props":1221,"children":1222},{"style":416},[1223],{"type":47,"value":1224},")\n",{"type":41,"tag":211,"props":1226,"children":1227},{"class":213,"line":603},[1228,1233,1238,1242],{"type":41,"tag":211,"props":1229,"children":1230},{"style":1157},[1231],{"type":47,"value":1232},"JOB_ID",{"type":41,"tag":211,"props":1234,"children":1235},{"style":416},[1236],{"type":47,"value":1237},"=\u003C",{"type":41,"tag":211,"props":1239,"children":1240},{"style":224},[1241],{"type":47,"value":473},{"type":41,"tag":211,"props":1243,"children":1244},{"style":416},[1245],{"type":47,"value":1246},">\n",{"type":41,"tag":211,"props":1248,"children":1249},{"class":213,"line":668},[1250,1255,1259,1263,1267,1272,1276,1281],{"type":41,"tag":211,"props":1251,"children":1252},{"style":1157},[1253],{"type":47,"value":1254},"CKPT_DIR",{"type":41,"tag":211,"props":1256,"children":1257},{"style":416},[1258],{"type":47,"value":1165},{"type":41,"tag":211,"props":1260,"children":1261},{"style":218},[1262],{"type":47,"value":253},{"type":41,"tag":211,"props":1264,"children":1265},{"style":224},[1266],{"type":47,"value":790},{"type":41,"tag":211,"props":1268,"children":1269},{"style":1157},[1270],{"type":47,"value":1271}," $CR_CONT ",{"type":41,"tag":211,"props":1273,"children":1274},{"style":224},[1275],{"type":47,"value":204},{"type":41,"tag":211,"props":1277,"children":1278},{"style":224},[1279],{"type":47,"value":1280}," -c",{"type":41,"tag":211,"props":1282,"children":1283},{"style":1157},[1284],{"type":47,"value":1285}," \\\n",{"type":41,"tag":211,"props":1287,"children":1288},{"class":213,"line":678},[1289,1294,1299,1304,1309,1313],{"type":41,"tag":211,"props":1290,"children":1291},{"style":416},[1292],{"type":47,"value":1293}," \"",{"type":41,"tag":211,"props":1295,"children":1296},{"style":224},[1297],{"type":47,"value":1298},"ls -d \u002Fworkspace\u002Fsop-cr-ftms\u002Fassets\u002Fresults\u002F",{"type":41,"tag":211,"props":1300,"children":1301},{"style":1157},[1302],{"type":47,"value":1303},"$JOB_ID",{"type":41,"tag":211,"props":1305,"children":1306},{"style":224},[1307],{"type":47,"value":1308},"\u002F*\u002Fsafetensors\u002Fstep_* 2>\u002Fdev\u002Fnull | sort -V | tail -1",{"type":41,"tag":211,"props":1310,"children":1311},{"style":416},[1312],{"type":47,"value":429},{"type":41,"tag":211,"props":1314,"children":1315},{"style":416},[1316],{"type":47,"value":1224},{"type":41,"tag":211,"props":1318,"children":1319},{"class":213,"line":687},[1320,1326,1330,1335,1340],{"type":41,"tag":211,"props":1321,"children":1323},{"style":1322},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1324],{"type":47,"value":1325},"echo",{"type":41,"tag":211,"props":1327,"children":1328},{"style":416},[1329],{"type":47,"value":419},{"type":41,"tag":211,"props":1331,"children":1332},{"style":224},[1333],{"type":47,"value":1334},"checkpoint dir: ",{"type":41,"tag":211,"props":1336,"children":1337},{"style":1157},[1338],{"type":47,"value":1339},"$CKPT_DIR",{"type":41,"tag":211,"props":1341,"children":1342},{"style":416},[1343],{"type":47,"value":448},{"type":41,"tag":211,"props":1345,"children":1346},{"class":213,"line":759},[1347],{"type":41,"tag":211,"props":1348,"children":1349},{"emptyLinePlaceholder":672},[1350],{"type":47,"value":675},{"type":41,"tag":211,"props":1352,"children":1353},{"class":213,"line":767},[1354,1360,1365,1369,1373,1378,1383,1387,1391,1396,1400,1405],{"type":41,"tag":211,"props":1355,"children":1357},{"style":1356},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[1358],{"type":47,"value":1359},"if",{"type":41,"tag":211,"props":1361,"children":1362},{"style":218},[1363],{"type":47,"value":1364}," docker",{"type":41,"tag":211,"props":1366,"children":1367},{"style":224},[1368],{"type":47,"value":790},{"type":41,"tag":211,"props":1370,"children":1371},{"style":1157},[1372],{"type":47,"value":1271},{"type":41,"tag":211,"props":1374,"children":1375},{"style":224},[1376],{"type":47,"value":1377},"test",{"type":41,"tag":211,"props":1379,"children":1380},{"style":224},[1381],{"type":47,"value":1382}," -f",{"type":41,"tag":211,"props":1384,"children":1385},{"style":416},[1386],{"type":47,"value":419},{"type":41,"tag":211,"props":1388,"children":1389},{"style":1157},[1390],{"type":47,"value":1339},{"type":41,"tag":211,"props":1392,"children":1393},{"style":224},[1394],{"type":47,"value":1395},"\u002Fadapter_config.json",{"type":41,"tag":211,"props":1397,"children":1398},{"style":416},[1399],{"type":47,"value":429},{"type":41,"tag":211,"props":1401,"children":1402},{"style":416},[1403],{"type":47,"value":1404},";",{"type":41,"tag":211,"props":1406,"children":1407},{"style":1356},[1408],{"type":47,"value":1409}," then\n",{"type":41,"tag":211,"props":1411,"children":1412},{"class":213,"line":776},[1413,1418,1422,1427],{"type":41,"tag":211,"props":1414,"children":1415},{"style":1322},[1416],{"type":47,"value":1417}," echo",{"type":41,"tag":211,"props":1419,"children":1420},{"style":416},[1421],{"type":47,"value":419},{"type":41,"tag":211,"props":1423,"children":1424},{"style":224},[1425],{"type":47,"value":1426},"LoRA adapter detected — run steps 3b.1 → 3b.3.",{"type":41,"tag":211,"props":1428,"children":1429},{"style":416},[1430],{"type":47,"value":448},{"type":41,"tag":211,"props":1432,"children":1434},{"class":213,"line":1433},9,[1435],{"type":41,"tag":211,"props":1436,"children":1437},{"style":1356},[1438],{"type":47,"value":1439},"else\n",{"type":41,"tag":211,"props":1441,"children":1443},{"class":213,"line":1442},10,[1444,1448,1452,1457],{"type":41,"tag":211,"props":1445,"children":1446},{"style":1322},[1447],{"type":47,"value":1417},{"type":41,"tag":211,"props":1449,"children":1450},{"style":416},[1451],{"type":47,"value":419},{"type":41,"tag":211,"props":1453,"children":1454},{"style":224},[1455],{"type":47,"value":1456},"Full fine-tune detected — skip 3b.1 → 3b.3. step_\u003CN>\u002F is already an inference-ready model.",{"type":41,"tag":211,"props":1458,"children":1459},{"style":416},[1460],{"type":47,"value":448},{"type":41,"tag":211,"props":1462,"children":1464},{"class":213,"line":1463},11,[1465],{"type":41,"tag":211,"props":1466,"children":1467},{"style":1356},[1468],{"type":47,"value":1469},"fi\n",{"type":41,"tag":50,"props":1471,"children":1472},{},[1473],{"type":47,"value":1474},"If full fine-tune: skip the rest of this section.",{"type":41,"tag":1476,"props":1477,"children":1479},"h5",{"id":1478},"_3b1-normalise-adapter_configjson-for-forward-compat",[1480,1482,1487],{"type":47,"value":1481},"3b.1 — Normalise ",{"type":41,"tag":61,"props":1483,"children":1485},{"className":1484},[],[1486],{"type":47,"value":1143},{"type":47,"value":1488}," for forward-compat",{"type":41,"tag":50,"props":1490,"children":1491},{},[1492,1494,1500,1502,1508,1510,1516],{"type":47,"value":1493},"Cosmos-RL's training stack may serialise LoRA config fields the inference-side peft doesn't accept (e.g. ",{"type":41,"tag":61,"props":1495,"children":1497},{"className":1496},[],[1498],{"type":47,"value":1499},"r_pattern",{"type":47,"value":1501},"), or write ",{"type":41,"tag":61,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":47,"value":1507},"null",{"type":47,"value":1509}," where a dict is expected (e.g. ",{"type":41,"tag":61,"props":1511,"children":1513},{"className":1512},[],[1514],{"type":47,"value":1515},"alpha_pattern: null",{"type":47,"value":1517},"). Normalise defensively before loading:",{"type":41,"tag":200,"props":1519,"children":1521},{"className":202,"code":1520,"language":204,"meta":205,"style":205},"docker exec $CR_CONT python3 -c \"\nimport json\np='$CKPT_DIR\u002Fadapter_config.json'\nd=json.load(open(p))\nd.pop('r_pattern', None) # drop unknown fields\nfor k in ('alpha_pattern','rank_pattern'):\n if d.get(k) is None: d[k] = {} # null -> {} (peft expects dict)\nfor k,v in dict(bias='none', task_type='CAUSAL_LM', inference_mode=True,\n fan_in_fan_out=False, layers_to_transform=None,\n layers_pattern=None, megatron_config=None,\n megatron_core='megatron.core', loftq_config={},\n use_dora=False, init_lora_weights=True).items():\n d.setdefault(k, v) # backfill defaults\njson.dump(d, open(p,'w'), indent=2)\nprint('adapter_config.json normalised')\n\"\n",[1522],{"type":41,"tag":61,"props":1523,"children":1524},{"__ignoreMap":205},[1525,1554,1562,1579,1587,1595,1603,1611,1619,1627,1635,1643,1651,1660,1669,1678],{"type":41,"tag":211,"props":1526,"children":1527},{"class":213,"line":214},[1528,1532,1536,1540,1545,1549],{"type":41,"tag":211,"props":1529,"children":1530},{"style":218},[1531],{"type":47,"value":253},{"type":41,"tag":211,"props":1533,"children":1534},{"style":224},[1535],{"type":47,"value":790},{"type":41,"tag":211,"props":1537,"children":1538},{"style":1157},[1539],{"type":47,"value":1271},{"type":41,"tag":211,"props":1541,"children":1542},{"style":224},[1543],{"type":47,"value":1544},"python3",{"type":41,"tag":211,"props":1546,"children":1547},{"style":224},[1548],{"type":47,"value":1280},{"type":41,"tag":211,"props":1550,"children":1551},{"style":416},[1552],{"type":47,"value":1553}," \"\n",{"type":41,"tag":211,"props":1555,"children":1556},{"class":213,"line":603},[1557],{"type":41,"tag":211,"props":1558,"children":1559},{"style":224},[1560],{"type":47,"value":1561},"import json\n",{"type":41,"tag":211,"props":1563,"children":1564},{"class":213,"line":668},[1565,1570,1574],{"type":41,"tag":211,"props":1566,"children":1567},{"style":224},[1568],{"type":47,"value":1569},"p='",{"type":41,"tag":211,"props":1571,"children":1572},{"style":1157},[1573],{"type":47,"value":1339},{"type":41,"tag":211,"props":1575,"children":1576},{"style":224},[1577],{"type":47,"value":1578},"\u002Fadapter_config.json'\n",{"type":41,"tag":211,"props":1580,"children":1581},{"class":213,"line":678},[1582],{"type":41,"tag":211,"props":1583,"children":1584},{"style":224},[1585],{"type":47,"value":1586},"d=json.load(open(p))\n",{"type":41,"tag":211,"props":1588,"children":1589},{"class":213,"line":687},[1590],{"type":41,"tag":211,"props":1591,"children":1592},{"style":224},[1593],{"type":47,"value":1594},"d.pop('r_pattern', None) # drop unknown fields\n",{"type":41,"tag":211,"props":1596,"children":1597},{"class":213,"line":759},[1598],{"type":41,"tag":211,"props":1599,"children":1600},{"style":224},[1601],{"type":47,"value":1602},"for k in ('alpha_pattern','rank_pattern'):\n",{"type":41,"tag":211,"props":1604,"children":1605},{"class":213,"line":767},[1606],{"type":41,"tag":211,"props":1607,"children":1608},{"style":224},[1609],{"type":47,"value":1610}," if d.get(k) is None: d[k] = {} # null -> {} (peft expects dict)\n",{"type":41,"tag":211,"props":1612,"children":1613},{"class":213,"line":776},[1614],{"type":41,"tag":211,"props":1615,"children":1616},{"style":224},[1617],{"type":47,"value":1618},"for k,v in dict(bias='none', task_type='CAUSAL_LM', inference_mode=True,\n",{"type":41,"tag":211,"props":1620,"children":1621},{"class":213,"line":1433},[1622],{"type":41,"tag":211,"props":1623,"children":1624},{"style":224},[1625],{"type":47,"value":1626}," fan_in_fan_out=False, layers_to_transform=None,\n",{"type":41,"tag":211,"props":1628,"children":1629},{"class":213,"line":1442},[1630],{"type":41,"tag":211,"props":1631,"children":1632},{"style":224},[1633],{"type":47,"value":1634}," layers_pattern=None, megatron_config=None,\n",{"type":41,"tag":211,"props":1636,"children":1637},{"class":213,"line":1463},[1638],{"type":41,"tag":211,"props":1639,"children":1640},{"style":224},[1641],{"type":47,"value":1642}," megatron_core='megatron.core', loftq_config={},\n",{"type":41,"tag":211,"props":1644,"children":1645},{"class":213,"line":27},[1646],{"type":41,"tag":211,"props":1647,"children":1648},{"style":224},[1649],{"type":47,"value":1650}," use_dora=False, init_lora_weights=True).items():\n",{"type":41,"tag":211,"props":1652,"children":1654},{"class":213,"line":1653},13,[1655],{"type":41,"tag":211,"props":1656,"children":1657},{"style":224},[1658],{"type":47,"value":1659}," d.setdefault(k, v) # backfill defaults\n",{"type":41,"tag":211,"props":1661,"children":1663},{"class":213,"line":1662},14,[1664],{"type":41,"tag":211,"props":1665,"children":1666},{"style":224},[1667],{"type":47,"value":1668},"json.dump(d, open(p,'w'), indent=2)\n",{"type":41,"tag":211,"props":1670,"children":1672},{"class":213,"line":1671},15,[1673],{"type":41,"tag":211,"props":1674,"children":1675},{"style":224},[1676],{"type":47,"value":1677},"print('adapter_config.json normalised')\n",{"type":41,"tag":211,"props":1679,"children":1681},{"class":213,"line":1680},16,[1682],{"type":41,"tag":211,"props":1683,"children":1684},{"style":416},[1685],{"type":47,"value":448},{"type":41,"tag":1476,"props":1687,"children":1689},{"id":1688},"_3b2-merge-adaptor-with-base-model",[1690],{"type":47,"value":1691},"3b.2 — Merge adaptor with base model",{"type":41,"tag":50,"props":1693,"children":1694},{},[1695,1697,1703],{"type":47,"value":1696},"Write to a sibling ",{"type":41,"tag":61,"props":1698,"children":1700},{"className":1699},[],[1701],{"type":47,"value":1702},"step_\u003CN>_merged_tmp\u002F",{"type":47,"value":1704}," so the final atomic swap (3b.3) is crash-safe:",{"type":41,"tag":50,"props":1706,"children":1707},{},[1708,1710,1715],{"type":47,"value":1709},"The base model path and the base model class are both ",{"type":41,"tag":129,"props":1711,"children":1712},{},[1713],{"type":47,"value":1714},"derived from the adapter",{"type":47,"value":1716}," at run-time:",{"type":41,"tag":121,"props":1718,"children":1719},{},[1720,1754],{"type":41,"tag":125,"props":1721,"children":1722},{},[1723,1728,1730,1736,1738,1744,1746,1752],{"type":41,"tag":129,"props":1724,"children":1725},{},[1726],{"type":47,"value":1727},"Base path",{"type":47,"value":1729}," comes from ",{"type":41,"tag":61,"props":1731,"children":1733},{"className":1732},[],[1734],{"type":47,"value":1735},"adapter_config.json[\"base_model_name_or_path\"]",{"type":47,"value":1737},", which peft writes at training time. It is typically a path relative to the training process's working directory (",{"type":41,"tag":61,"props":1739,"children":1741},{"className":1740},[],[1742],{"type":47,"value":1743},"\u002Fworkspace\u002Fsop-cr-ftms",{"type":47,"value":1745},"), e.g. ",{"type":41,"tag":61,"props":1747,"children":1749},{"className":1748},[],[1750],{"type":47,"value":1751},".\u002Fassets\u002Fweights\u002FCosmos-Reason2-2B",{"type":47,"value":1753},". Resolve it against that working dir if it isn't already absolute.",{"type":41,"tag":125,"props":1755,"children":1756},{},[1757,1762,1763,1769,1771,1777,1779,1785,1787,1793,1795,1801],{"type":41,"tag":129,"props":1758,"children":1759},{},[1760],{"type":47,"value":1761},"Model class",{"type":47,"value":1729},{"type":41,"tag":61,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":47,"value":1768},"AutoModelForImageTextToText",{"type":47,"value":1770},", which reads ",{"type":41,"tag":61,"props":1772,"children":1774},{"className":1773},[],[1775],{"type":47,"value":1776},"architectures",{"type":47,"value":1778}," from the base model's ",{"type":41,"tag":61,"props":1780,"children":1782},{"className":1781},[],[1783],{"type":47,"value":1784},"config.json",{"type":47,"value":1786}," and instantiates the right class (e.g. ",{"type":41,"tag":61,"props":1788,"children":1790},{"className":1789},[],[1791],{"type":47,"value":1792},"Qwen3VLForConditionalGeneration",{"type":47,"value":1794}," for Cosmos-Reason2-2B, ",{"type":41,"tag":61,"props":1796,"children":1798},{"className":1797},[],[1799],{"type":47,"value":1800},"Qwen2_5_VLForConditionalGeneration",{"type":47,"value":1802}," for older variants).",{"type":41,"tag":200,"props":1804,"children":1806},{"className":202,"code":1805,"language":204,"meta":205,"style":205},"docker exec -w \u002Fworkspace\u002Fsop-cr-ftms $CR_CONT python3 -c \"\nimport json, os, torch\nfrom transformers import AutoModelForImageTextToText, AutoProcessor\nfrom peft import PeftModel\n\nadapter_path = '$CKPT_DIR'\ntmp_path = adapter_path + '_merged_tmp'\n\n# Resolve the base model path from the adapter's own metadata.\nwith open(os.path.join(adapter_path, 'adapter_config.json')) as f:\n base_rel = json.load(f)['base_model_name_or_path']\nbase_path = (base_rel if os.path.isabs(base_rel)\n else os.path.normpath(os.path.join('\u002Fworkspace\u002Fsop-cr-ftms', base_rel)))\nprint(f'base model: {base_path}', flush=True)\n\nprint('loading base...', flush=True)\nbase = AutoModelForImageTextToText.from_pretrained(\n base_path, dtype=torch.bfloat16, device_map='cpu')\nprint('loading adapter...', flush=True)\nmodel = PeftModel.from_pretrained(base, adapter_path)\nprint('merging...', flush=True)\nmerged = model.merge_and_unload()\n\nos.makedirs(tmp_path, exist_ok=True)\nmerged.save_pretrained(tmp_path, safe_serialization=True)\n# Processor\u002Fpreprocessor configs live with the base model, not the adapter.\n# The adapter dir only carries tokenizer files.\nAutoProcessor.from_pretrained(base_path).save_pretrained(tmp_path)\nprint('merged ->', tmp_path)\n\"\n",[1807],{"type":41,"tag":61,"props":1808,"children":1809},{"__ignoreMap":205},[1810,1847,1855,1863,1871,1878,1895,1903,1910,1918,1926,1934,1942,1950,1958,1965,1973,1982,1991,2000,2009,2018,2027,2035,2044,2053,2062,2071,2080,2089],{"type":41,"tag":211,"props":1811,"children":1812},{"class":213,"line":214},[1813,1817,1821,1826,1831,1835,1839,1843],{"type":41,"tag":211,"props":1814,"children":1815},{"style":218},[1816],{"type":47,"value":253},{"type":41,"tag":211,"props":1818,"children":1819},{"style":224},[1820],{"type":47,"value":790},{"type":41,"tag":211,"props":1822,"children":1823},{"style":224},[1824],{"type":47,"value":1825}," -w",{"type":41,"tag":211,"props":1827,"children":1828},{"style":224},[1829],{"type":47,"value":1830}," \u002Fworkspace\u002Fsop-cr-ftms",{"type":41,"tag":211,"props":1832,"children":1833},{"style":1157},[1834],{"type":47,"value":1271},{"type":41,"tag":211,"props":1836,"children":1837},{"style":224},[1838],{"type":47,"value":1544},{"type":41,"tag":211,"props":1840,"children":1841},{"style":224},[1842],{"type":47,"value":1280},{"type":41,"tag":211,"props":1844,"children":1845},{"style":416},[1846],{"type":47,"value":1553},{"type":41,"tag":211,"props":1848,"children":1849},{"class":213,"line":603},[1850],{"type":41,"tag":211,"props":1851,"children":1852},{"style":224},[1853],{"type":47,"value":1854},"import json, os, torch\n",{"type":41,"tag":211,"props":1856,"children":1857},{"class":213,"line":668},[1858],{"type":41,"tag":211,"props":1859,"children":1860},{"style":224},[1861],{"type":47,"value":1862},"from transformers import AutoModelForImageTextToText, AutoProcessor\n",{"type":41,"tag":211,"props":1864,"children":1865},{"class":213,"line":678},[1866],{"type":41,"tag":211,"props":1867,"children":1868},{"style":224},[1869],{"type":47,"value":1870},"from peft import PeftModel\n",{"type":41,"tag":211,"props":1872,"children":1873},{"class":213,"line":687},[1874],{"type":41,"tag":211,"props":1875,"children":1876},{"emptyLinePlaceholder":672},[1877],{"type":47,"value":675},{"type":41,"tag":211,"props":1879,"children":1880},{"class":213,"line":759},[1881,1886,1890],{"type":41,"tag":211,"props":1882,"children":1883},{"style":224},[1884],{"type":47,"value":1885},"adapter_path = '",{"type":41,"tag":211,"props":1887,"children":1888},{"style":1157},[1889],{"type":47,"value":1339},{"type":41,"tag":211,"props":1891,"children":1892},{"style":224},[1893],{"type":47,"value":1894},"'\n",{"type":41,"tag":211,"props":1896,"children":1897},{"class":213,"line":767},[1898],{"type":41,"tag":211,"props":1899,"children":1900},{"style":224},[1901],{"type":47,"value":1902},"tmp_path = adapter_path + '_merged_tmp'\n",{"type":41,"tag":211,"props":1904,"children":1905},{"class":213,"line":776},[1906],{"type":41,"tag":211,"props":1907,"children":1908},{"emptyLinePlaceholder":672},[1909],{"type":47,"value":675},{"type":41,"tag":211,"props":1911,"children":1912},{"class":213,"line":1433},[1913],{"type":41,"tag":211,"props":1914,"children":1915},{"style":224},[1916],{"type":47,"value":1917},"# Resolve the base model path from the adapter's own metadata.\n",{"type":41,"tag":211,"props":1919,"children":1920},{"class":213,"line":1442},[1921],{"type":41,"tag":211,"props":1922,"children":1923},{"style":224},[1924],{"type":47,"value":1925},"with open(os.path.join(adapter_path, 'adapter_config.json')) as f:\n",{"type":41,"tag":211,"props":1927,"children":1928},{"class":213,"line":1463},[1929],{"type":41,"tag":211,"props":1930,"children":1931},{"style":224},[1932],{"type":47,"value":1933}," base_rel = json.load(f)['base_model_name_or_path']\n",{"type":41,"tag":211,"props":1935,"children":1936},{"class":213,"line":27},[1937],{"type":41,"tag":211,"props":1938,"children":1939},{"style":224},[1940],{"type":47,"value":1941},"base_path = (base_rel if os.path.isabs(base_rel)\n",{"type":41,"tag":211,"props":1943,"children":1944},{"class":213,"line":1653},[1945],{"type":41,"tag":211,"props":1946,"children":1947},{"style":224},[1948],{"type":47,"value":1949}," else os.path.normpath(os.path.join('\u002Fworkspace\u002Fsop-cr-ftms', base_rel)))\n",{"type":41,"tag":211,"props":1951,"children":1952},{"class":213,"line":1662},[1953],{"type":41,"tag":211,"props":1954,"children":1955},{"style":224},[1956],{"type":47,"value":1957},"print(f'base model: {base_path}', flush=True)\n",{"type":41,"tag":211,"props":1959,"children":1960},{"class":213,"line":1671},[1961],{"type":41,"tag":211,"props":1962,"children":1963},{"emptyLinePlaceholder":672},[1964],{"type":47,"value":675},{"type":41,"tag":211,"props":1966,"children":1967},{"class":213,"line":1680},[1968],{"type":41,"tag":211,"props":1969,"children":1970},{"style":224},[1971],{"type":47,"value":1972},"print('loading base...', flush=True)\n",{"type":41,"tag":211,"props":1974,"children":1976},{"class":213,"line":1975},17,[1977],{"type":41,"tag":211,"props":1978,"children":1979},{"style":224},[1980],{"type":47,"value":1981},"base = AutoModelForImageTextToText.from_pretrained(\n",{"type":41,"tag":211,"props":1983,"children":1985},{"class":213,"line":1984},18,[1986],{"type":41,"tag":211,"props":1987,"children":1988},{"style":224},[1989],{"type":47,"value":1990}," base_path, dtype=torch.bfloat16, device_map='cpu')\n",{"type":41,"tag":211,"props":1992,"children":1994},{"class":213,"line":1993},19,[1995],{"type":41,"tag":211,"props":1996,"children":1997},{"style":224},[1998],{"type":47,"value":1999},"print('loading adapter...', flush=True)\n",{"type":41,"tag":211,"props":2001,"children":2003},{"class":213,"line":2002},20,[2004],{"type":41,"tag":211,"props":2005,"children":2006},{"style":224},[2007],{"type":47,"value":2008},"model = PeftModel.from_pretrained(base, adapter_path)\n",{"type":41,"tag":211,"props":2010,"children":2012},{"class":213,"line":2011},21,[2013],{"type":41,"tag":211,"props":2014,"children":2015},{"style":224},[2016],{"type":47,"value":2017},"print('merging...', flush=True)\n",{"type":41,"tag":211,"props":2019,"children":2021},{"class":213,"line":2020},22,[2022],{"type":41,"tag":211,"props":2023,"children":2024},{"style":224},[2025],{"type":47,"value":2026},"merged = model.merge_and_unload()\n",{"type":41,"tag":211,"props":2028,"children":2030},{"class":213,"line":2029},23,[2031],{"type":41,"tag":211,"props":2032,"children":2033},{"emptyLinePlaceholder":672},[2034],{"type":47,"value":675},{"type":41,"tag":211,"props":2036,"children":2038},{"class":213,"line":2037},24,[2039],{"type":41,"tag":211,"props":2040,"children":2041},{"style":224},[2042],{"type":47,"value":2043},"os.makedirs(tmp_path, exist_ok=True)\n",{"type":41,"tag":211,"props":2045,"children":2047},{"class":213,"line":2046},25,[2048],{"type":41,"tag":211,"props":2049,"children":2050},{"style":224},[2051],{"type":47,"value":2052},"merged.save_pretrained(tmp_path, safe_serialization=True)\n",{"type":41,"tag":211,"props":2054,"children":2056},{"class":213,"line":2055},26,[2057],{"type":41,"tag":211,"props":2058,"children":2059},{"style":224},[2060],{"type":47,"value":2061},"# Processor\u002Fpreprocessor configs live with the base model, not the adapter.\n",{"type":41,"tag":211,"props":2063,"children":2065},{"class":213,"line":2064},27,[2066],{"type":41,"tag":211,"props":2067,"children":2068},{"style":224},[2069],{"type":47,"value":2070},"# The adapter dir only carries tokenizer files.\n",{"type":41,"tag":211,"props":2072,"children":2074},{"class":213,"line":2073},28,[2075],{"type":41,"tag":211,"props":2076,"children":2077},{"style":224},[2078],{"type":47,"value":2079},"AutoProcessor.from_pretrained(base_path).save_pretrained(tmp_path)\n",{"type":41,"tag":211,"props":2081,"children":2083},{"class":213,"line":2082},29,[2084],{"type":41,"tag":211,"props":2085,"children":2086},{"style":224},[2087],{"type":47,"value":2088},"print('merged ->', tmp_path)\n",{"type":41,"tag":211,"props":2090,"children":2092},{"class":213,"line":2091},30,[2093],{"type":41,"tag":211,"props":2094,"children":2095},{"style":416},[2096],{"type":47,"value":448},{"type":41,"tag":50,"props":2098,"children":2099},{},[2100],{"type":47,"value":2101},"Notes on the kwargs:",{"type":41,"tag":121,"props":2103,"children":2104},{},[2105,2116,2127],{"type":41,"tag":125,"props":2106,"children":2107},{},[2108,2114],{"type":41,"tag":61,"props":2109,"children":2111},{"className":2110},[],[2112],{"type":47,"value":2113},"device_map='cpu'",{"type":47,"value":2115}," — merge is memory-bandwidth-bound and uses ~6 GB CPU RAM. Keeps GPUs free for concurrent training\u002Feval.",{"type":41,"tag":125,"props":2117,"children":2118},{},[2119,2125],{"type":41,"tag":61,"props":2120,"children":2122},{"className":2121},[],[2123],{"type":47,"value":2124},"dtype=torch.bfloat16",{"type":47,"value":2126}," — matches the precision the LoRA was trained against.",{"type":41,"tag":125,"props":2128,"children":2129},{},[2130,2132,2138,2140,2145],{"type":47,"value":2131},"The processor (",{"type":41,"tag":61,"props":2133,"children":2135},{"className":2134},[],[2136],{"type":47,"value":2137},"AutoProcessor",{"type":47,"value":2139},") is loaded from the ",{"type":41,"tag":129,"props":2141,"children":2142},{},[2143],{"type":47,"value":2144},"base",{"type":47,"value":2146}," model path, not the adapter path. Vision\u002Fvideo preprocessor configs live with the base; the adapter dir only carries tokenizer + adapter files.",{"type":41,"tag":1476,"props":2148,"children":2150},{"id":2149},"_3b3-atomic-swap-into-step_n",[2151,2153],{"type":47,"value":2152},"3b.3 — Atomic swap into ",{"type":41,"tag":61,"props":2154,"children":2156},{"className":2155},[],[2157],{"type":47,"value":1000},{"type":41,"tag":50,"props":2159,"children":2160},{},[2161,2163,2169,2171,2177,2179,2185],{"type":47,"value":2162},"Evaluation-ms resolves checkpoints by ",{"type":41,"tag":61,"props":2164,"children":2166},{"className":2165},[],[2167],{"type":47,"value":2168},"safetensors\u002Fstep_\u003CN>\u002F",{"type":47,"value":2170}," and calls ",{"type":41,"tag":61,"props":2172,"children":2174},{"className":2173},[],[2175],{"type":47,"value":2176},"from_pretrained",{"type":47,"value":2178}," on that exact path; it has no knowledge of ",{"type":41,"tag":61,"props":2180,"children":2182},{"className":2181},[],[2183],{"type":47,"value":2184},"merged_model\u002F",{"type":47,"value":2186}," or other subdirs. Swap the merged result in and keep the original adapter as a sibling:",{"type":41,"tag":200,"props":2188,"children":2190},{"className":202,"code":2189,"language":204,"meta":205,"style":205},"docker exec $CR_CONT bash -c \"\n STEP_DIR='$CKPT_DIR'\n PARENT=\\$(dirname \\$STEP_DIR)\n STEP=\\$(basename \\$STEP_DIR)\n mv \\$STEP_DIR \\${PARENT}\u002F\\${STEP}_lora_adapter\n mv \\${PARENT}\u002F\\${STEP}_merged_tmp \\$STEP_DIR\n\"\n",[2191],{"type":41,"tag":61,"props":2192,"children":2193},{"__ignoreMap":205},[2194,2221,2237,2264,2289,2324,2357],{"type":41,"tag":211,"props":2195,"children":2196},{"class":213,"line":214},[2197,2201,2205,2209,2213,2217],{"type":41,"tag":211,"props":2198,"children":2199},{"style":218},[2200],{"type":47,"value":253},{"type":41,"tag":211,"props":2202,"children":2203},{"style":224},[2204],{"type":47,"value":790},{"type":41,"tag":211,"props":2206,"children":2207},{"style":1157},[2208],{"type":47,"value":1271},{"type":41,"tag":211,"props":2210,"children":2211},{"style":224},[2212],{"type":47,"value":204},{"type":41,"tag":211,"props":2214,"children":2215},{"style":224},[2216],{"type":47,"value":1280},{"type":41,"tag":211,"props":2218,"children":2219},{"style":416},[2220],{"type":47,"value":1553},{"type":41,"tag":211,"props":2222,"children":2223},{"class":213,"line":603},[2224,2229,2233],{"type":41,"tag":211,"props":2225,"children":2226},{"style":224},[2227],{"type":47,"value":2228}," STEP_DIR='",{"type":41,"tag":211,"props":2230,"children":2231},{"style":1157},[2232],{"type":47,"value":1339},{"type":41,"tag":211,"props":2234,"children":2235},{"style":224},[2236],{"type":47,"value":1894},{"type":41,"tag":211,"props":2238,"children":2239},{"class":213,"line":668},[2240,2245,2250,2255,2259],{"type":41,"tag":211,"props":2241,"children":2242},{"style":224},[2243],{"type":47,"value":2244}," PARENT=",{"type":41,"tag":211,"props":2246,"children":2247},{"style":1157},[2248],{"type":47,"value":2249},"\\$",{"type":41,"tag":211,"props":2251,"children":2252},{"style":224},[2253],{"type":47,"value":2254},"(dirname ",{"type":41,"tag":211,"props":2256,"children":2257},{"style":1157},[2258],{"type":47,"value":2249},{"type":41,"tag":211,"props":2260,"children":2261},{"style":224},[2262],{"type":47,"value":2263},"STEP_DIR)\n",{"type":41,"tag":211,"props":2265,"children":2266},{"class":213,"line":678},[2267,2272,2276,2281,2285],{"type":41,"tag":211,"props":2268,"children":2269},{"style":224},[2270],{"type":47,"value":2271}," STEP=",{"type":41,"tag":211,"props":2273,"children":2274},{"style":1157},[2275],{"type":47,"value":2249},{"type":41,"tag":211,"props":2277,"children":2278},{"style":224},[2279],{"type":47,"value":2280},"(basename ",{"type":41,"tag":211,"props":2282,"children":2283},{"style":1157},[2284],{"type":47,"value":2249},{"type":41,"tag":211,"props":2286,"children":2287},{"style":224},[2288],{"type":47,"value":2263},{"type":41,"tag":211,"props":2290,"children":2291},{"class":213,"line":687},[2292,2297,2301,2306,2310,2315,2319],{"type":41,"tag":211,"props":2293,"children":2294},{"style":224},[2295],{"type":47,"value":2296}," mv ",{"type":41,"tag":211,"props":2298,"children":2299},{"style":1157},[2300],{"type":47,"value":2249},{"type":41,"tag":211,"props":2302,"children":2303},{"style":224},[2304],{"type":47,"value":2305},"STEP_DIR ",{"type":41,"tag":211,"props":2307,"children":2308},{"style":1157},[2309],{"type":47,"value":2249},{"type":41,"tag":211,"props":2311,"children":2312},{"style":224},[2313],{"type":47,"value":2314},"{PARENT}\u002F",{"type":41,"tag":211,"props":2316,"children":2317},{"style":1157},[2318],{"type":47,"value":2249},{"type":41,"tag":211,"props":2320,"children":2321},{"style":224},[2322],{"type":47,"value":2323},"{STEP}_lora_adapter\n",{"type":41,"tag":211,"props":2325,"children":2326},{"class":213,"line":759},[2327,2331,2335,2339,2343,2348,2352],{"type":41,"tag":211,"props":2328,"children":2329},{"style":224},[2330],{"type":47,"value":2296},{"type":41,"tag":211,"props":2332,"children":2333},{"style":1157},[2334],{"type":47,"value":2249},{"type":41,"tag":211,"props":2336,"children":2337},{"style":224},[2338],{"type":47,"value":2314},{"type":41,"tag":211,"props":2340,"children":2341},{"style":1157},[2342],{"type":47,"value":2249},{"type":41,"tag":211,"props":2344,"children":2345},{"style":224},[2346],{"type":47,"value":2347},"{STEP}_merged_tmp ",{"type":41,"tag":211,"props":2349,"children":2350},{"style":1157},[2351],{"type":47,"value":2249},{"type":41,"tag":211,"props":2353,"children":2354},{"style":224},[2355],{"type":47,"value":2356},"STEP_DIR\n",{"type":41,"tag":211,"props":2358,"children":2359},{"class":213,"line":767},[2360],{"type":41,"tag":211,"props":2361,"children":2362},{"style":416},[2363],{"type":47,"value":448},{"type":41,"tag":1476,"props":2365,"children":2367},{"id":2366},"_3b4-verify",[2368],{"type":47,"value":2369},"3b.4 — Verify",{"type":41,"tag":50,"props":2371,"children":2372},{},[2373,2375,2380],{"type":47,"value":2374},"Listing the new ",{"type":41,"tag":61,"props":2376,"children":2378},{"className":2377},[],[2379],{"type":47,"value":1000},{"type":47,"value":2381}," should show full-model files, not adapter files:",{"type":41,"tag":200,"props":2383,"children":2385},{"className":202,"code":2384,"language":204,"meta":205,"style":205},"docker exec $CR_CONT ls \"$CKPT_DIR\"\n",[2386],{"type":41,"tag":61,"props":2387,"children":2388},{"__ignoreMap":205},[2389],{"type":41,"tag":211,"props":2390,"children":2391},{"class":213,"line":214},[2392,2396,2400,2404,2409,2413,2417],{"type":41,"tag":211,"props":2393,"children":2394},{"style":218},[2395],{"type":47,"value":253},{"type":41,"tag":211,"props":2397,"children":2398},{"style":224},[2399],{"type":47,"value":790},{"type":41,"tag":211,"props":2401,"children":2402},{"style":1157},[2403],{"type":47,"value":1271},{"type":41,"tag":211,"props":2405,"children":2406},{"style":224},[2407],{"type":47,"value":2408},"ls",{"type":41,"tag":211,"props":2410,"children":2411},{"style":416},[2412],{"type":47,"value":419},{"type":41,"tag":211,"props":2414,"children":2415},{"style":1157},[2416],{"type":47,"value":1339},{"type":41,"tag":211,"props":2418,"children":2419},{"style":416},[2420],{"type":47,"value":448},{"type":41,"tag":50,"props":2422,"children":2423},{},[2424,2426,2431,2432,2438,2440,2446,2448,2454,2455,2461,2462,2468,2470,2476],{"type":47,"value":2425},"Expect ",{"type":41,"tag":61,"props":2427,"children":2429},{"className":2428},[],[2430],{"type":47,"value":1784},{"type":47,"value":854},{"type":41,"tag":61,"props":2433,"children":2435},{"className":2434},[],[2436],{"type":47,"value":2437},"model.safetensors",{"type":47,"value":2439}," (or sharded ",{"type":41,"tag":61,"props":2441,"children":2443},{"className":2442},[],[2444],{"type":47,"value":2445},"model-00001-of-NNNNN.safetensors",{"type":47,"value":2447},"), ",{"type":41,"tag":61,"props":2449,"children":2451},{"className":2450},[],[2452],{"type":47,"value":2453},"tokenizer.json",{"type":47,"value":854},{"type":41,"tag":61,"props":2456,"children":2458},{"className":2457},[],[2459],{"type":47,"value":2460},"preprocessor_config.json",{"type":47,"value":854},{"type":41,"tag":61,"props":2463,"children":2465},{"className":2464},[],[2466],{"type":47,"value":2467},"video_preprocessor_config.json",{"type":47,"value":2469},", generation\u002Fspecial-token configs. The presence of ",{"type":41,"tag":61,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":47,"value":2475},"adapter_model.safetensors",{"type":47,"value":2477}," here means the merge did not land — recheck 3b.3.",{"type":41,"tag":891,"props":2479,"children":2481},{"id":2480},"_3c-write-training-report",[2482],{"type":47,"value":2483},"3c: Write Training Report",{"type":41,"tag":50,"props":2485,"children":2486},{},[2487,2489,2494],{"type":47,"value":2488},"Save a formatted training report to ",{"type":41,"tag":61,"props":2490,"children":2492},{"className":2491},[],[2493],{"type":47,"value":887},{"type":47,"value":198},{"type":41,"tag":200,"props":2496,"children":2500},{"className":2497,"code":2498,"language":2499,"meta":205,"style":205},"language-markdown shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Training Report\n\n## Overview\n| Field | Value |\n|-------|-------|\n| Dataset | \u003Cqa_augmented_dataset_id> |\n| Job ID | \u003Cjob_id> |\n| Status | completed \u002F failed \u002F cancelled |\n| Duration | Xh Xm |\n\n## Training Metrics\n| Metric | Value |\n|--------|-------|\n| Final Loss | X.XX |\n| Best Loss | X.XX (step N) |\n| Total Steps | N \u002F N |\n| Steps\u002Fsec | X.XX |\n| GPU Memory Peak | XX% |\n\n## Anomalies\n- List any OOM events, loss spikes, NaN occurrences, or GPU warnings detected during training.\n- \"None\" if no anomalies were detected.\n\n## Artifacts\n- **Inference-ready checkpoint:** `assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>\u002F` — always, regardless of training mode (Phase 3b merges LoRA into the base in place; full fine-tune writes here directly).\n- **LoRA adapter backup (LoRA runs only):** `assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>_lora_adapter\u002F` — the unmerged adapter, kept for reproducibility\u002Faudit.\n- **Training log:** `results\u002F\u003Cjob_id>\u002Flog.txt`\n","markdown",[2501],{"type":41,"tag":61,"props":2502,"children":2503},{"__ignoreMap":205},[2504,2517,2524,2537,2564,2572,2597,2622,2647,2672,2679,2691,2715,2723,2748,2773,2798,2822,2847,2854,2866,2879,2891,2898,2910,2953,2991],{"type":41,"tag":211,"props":2505,"children":2506},{"class":213,"line":214},[2507,2512],{"type":41,"tag":211,"props":2508,"children":2509},{"style":416},[2510],{"type":47,"value":2511},"# ",{"type":41,"tag":211,"props":2513,"children":2514},{"style":218},[2515],{"type":47,"value":2516},"Training Report\n",{"type":41,"tag":211,"props":2518,"children":2519},{"class":213,"line":603},[2520],{"type":41,"tag":211,"props":2521,"children":2522},{"emptyLinePlaceholder":672},[2523],{"type":47,"value":675},{"type":41,"tag":211,"props":2525,"children":2526},{"class":213,"line":668},[2527,2532],{"type":41,"tag":211,"props":2528,"children":2529},{"style":416},[2530],{"type":47,"value":2531},"## ",{"type":41,"tag":211,"props":2533,"children":2534},{"style":218},[2535],{"type":47,"value":2536},"Overview\n",{"type":41,"tag":211,"props":2538,"children":2539},{"class":213,"line":678},[2540,2545,2550,2554,2559],{"type":41,"tag":211,"props":2541,"children":2542},{"style":416},[2543],{"type":47,"value":2544},"|",{"type":41,"tag":211,"props":2546,"children":2547},{"style":1157},[2548],{"type":47,"value":2549}," Field ",{"type":41,"tag":211,"props":2551,"children":2552},{"style":416},[2553],{"type":47,"value":2544},{"type":41,"tag":211,"props":2555,"children":2556},{"style":1157},[2557],{"type":47,"value":2558}," Value ",{"type":41,"tag":211,"props":2560,"children":2561},{"style":416},[2562],{"type":47,"value":2563},"|\n",{"type":41,"tag":211,"props":2565,"children":2566},{"class":213,"line":687},[2567],{"type":41,"tag":211,"props":2568,"children":2569},{"style":416},[2570],{"type":47,"value":2571},"|-------|-------|\n",{"type":41,"tag":211,"props":2573,"children":2574},{"class":213,"line":759},[2575,2579,2584,2588,2593],{"type":41,"tag":211,"props":2576,"children":2577},{"style":416},[2578],{"type":47,"value":2544},{"type":41,"tag":211,"props":2580,"children":2581},{"style":1157},[2582],{"type":47,"value":2583}," Dataset ",{"type":41,"tag":211,"props":2585,"children":2586},{"style":416},[2587],{"type":47,"value":2544},{"type":41,"tag":211,"props":2589,"children":2590},{"style":1157},[2591],{"type":47,"value":2592}," \u003Cqa_augmented_dataset_id> ",{"type":41,"tag":211,"props":2594,"children":2595},{"style":416},[2596],{"type":47,"value":2563},{"type":41,"tag":211,"props":2598,"children":2599},{"class":213,"line":767},[2600,2604,2609,2613,2618],{"type":41,"tag":211,"props":2601,"children":2602},{"style":416},[2603],{"type":47,"value":2544},{"type":41,"tag":211,"props":2605,"children":2606},{"style":1157},[2607],{"type":47,"value":2608}," Job ID ",{"type":41,"tag":211,"props":2610,"children":2611},{"style":416},[2612],{"type":47,"value":2544},{"type":41,"tag":211,"props":2614,"children":2615},{"style":1157},[2616],{"type":47,"value":2617}," \u003Cjob_id> ",{"type":41,"tag":211,"props":2619,"children":2620},{"style":416},[2621],{"type":47,"value":2563},{"type":41,"tag":211,"props":2623,"children":2624},{"class":213,"line":776},[2625,2629,2634,2638,2643],{"type":41,"tag":211,"props":2626,"children":2627},{"style":416},[2628],{"type":47,"value":2544},{"type":41,"tag":211,"props":2630,"children":2631},{"style":1157},[2632],{"type":47,"value":2633}," Status ",{"type":41,"tag":211,"props":2635,"children":2636},{"style":416},[2637],{"type":47,"value":2544},{"type":41,"tag":211,"props":2639,"children":2640},{"style":1157},[2641],{"type":47,"value":2642}," completed \u002F failed \u002F cancelled ",{"type":41,"tag":211,"props":2644,"children":2645},{"style":416},[2646],{"type":47,"value":2563},{"type":41,"tag":211,"props":2648,"children":2649},{"class":213,"line":1433},[2650,2654,2659,2663,2668],{"type":41,"tag":211,"props":2651,"children":2652},{"style":416},[2653],{"type":47,"value":2544},{"type":41,"tag":211,"props":2655,"children":2656},{"style":1157},[2657],{"type":47,"value":2658}," Duration ",{"type":41,"tag":211,"props":2660,"children":2661},{"style":416},[2662],{"type":47,"value":2544},{"type":41,"tag":211,"props":2664,"children":2665},{"style":1157},[2666],{"type":47,"value":2667}," Xh Xm ",{"type":41,"tag":211,"props":2669,"children":2670},{"style":416},[2671],{"type":47,"value":2563},{"type":41,"tag":211,"props":2673,"children":2674},{"class":213,"line":1442},[2675],{"type":41,"tag":211,"props":2676,"children":2677},{"emptyLinePlaceholder":672},[2678],{"type":47,"value":675},{"type":41,"tag":211,"props":2680,"children":2681},{"class":213,"line":1463},[2682,2686],{"type":41,"tag":211,"props":2683,"children":2684},{"style":416},[2685],{"type":47,"value":2531},{"type":41,"tag":211,"props":2687,"children":2688},{"style":218},[2689],{"type":47,"value":2690},"Training Metrics\n",{"type":41,"tag":211,"props":2692,"children":2693},{"class":213,"line":27},[2694,2698,2703,2707,2711],{"type":41,"tag":211,"props":2695,"children":2696},{"style":416},[2697],{"type":47,"value":2544},{"type":41,"tag":211,"props":2699,"children":2700},{"style":1157},[2701],{"type":47,"value":2702}," Metric ",{"type":41,"tag":211,"props":2704,"children":2705},{"style":416},[2706],{"type":47,"value":2544},{"type":41,"tag":211,"props":2708,"children":2709},{"style":1157},[2710],{"type":47,"value":2558},{"type":41,"tag":211,"props":2712,"children":2713},{"style":416},[2714],{"type":47,"value":2563},{"type":41,"tag":211,"props":2716,"children":2717},{"class":213,"line":1653},[2718],{"type":41,"tag":211,"props":2719,"children":2720},{"style":416},[2721],{"type":47,"value":2722},"|--------|-------|\n",{"type":41,"tag":211,"props":2724,"children":2725},{"class":213,"line":1662},[2726,2730,2735,2739,2744],{"type":41,"tag":211,"props":2727,"children":2728},{"style":416},[2729],{"type":47,"value":2544},{"type":41,"tag":211,"props":2731,"children":2732},{"style":1157},[2733],{"type":47,"value":2734}," Final Loss ",{"type":41,"tag":211,"props":2736,"children":2737},{"style":416},[2738],{"type":47,"value":2544},{"type":41,"tag":211,"props":2740,"children":2741},{"style":1157},[2742],{"type":47,"value":2743}," X.XX ",{"type":41,"tag":211,"props":2745,"children":2746},{"style":416},[2747],{"type":47,"value":2563},{"type":41,"tag":211,"props":2749,"children":2750},{"class":213,"line":1671},[2751,2755,2760,2764,2769],{"type":41,"tag":211,"props":2752,"children":2753},{"style":416},[2754],{"type":47,"value":2544},{"type":41,"tag":211,"props":2756,"children":2757},{"style":1157},[2758],{"type":47,"value":2759}," Best Loss ",{"type":41,"tag":211,"props":2761,"children":2762},{"style":416},[2763],{"type":47,"value":2544},{"type":41,"tag":211,"props":2765,"children":2766},{"style":1157},[2767],{"type":47,"value":2768}," X.XX (step N) ",{"type":41,"tag":211,"props":2770,"children":2771},{"style":416},[2772],{"type":47,"value":2563},{"type":41,"tag":211,"props":2774,"children":2775},{"class":213,"line":1680},[2776,2780,2785,2789,2794],{"type":41,"tag":211,"props":2777,"children":2778},{"style":416},[2779],{"type":47,"value":2544},{"type":41,"tag":211,"props":2781,"children":2782},{"style":1157},[2783],{"type":47,"value":2784}," Total Steps ",{"type":41,"tag":211,"props":2786,"children":2787},{"style":416},[2788],{"type":47,"value":2544},{"type":41,"tag":211,"props":2790,"children":2791},{"style":1157},[2792],{"type":47,"value":2793}," N \u002F N ",{"type":41,"tag":211,"props":2795,"children":2796},{"style":416},[2797],{"type":47,"value":2563},{"type":41,"tag":211,"props":2799,"children":2800},{"class":213,"line":1975},[2801,2805,2810,2814,2818],{"type":41,"tag":211,"props":2802,"children":2803},{"style":416},[2804],{"type":47,"value":2544},{"type":41,"tag":211,"props":2806,"children":2807},{"style":1157},[2808],{"type":47,"value":2809}," Steps\u002Fsec ",{"type":41,"tag":211,"props":2811,"children":2812},{"style":416},[2813],{"type":47,"value":2544},{"type":41,"tag":211,"props":2815,"children":2816},{"style":1157},[2817],{"type":47,"value":2743},{"type":41,"tag":211,"props":2819,"children":2820},{"style":416},[2821],{"type":47,"value":2563},{"type":41,"tag":211,"props":2823,"children":2824},{"class":213,"line":1984},[2825,2829,2834,2838,2843],{"type":41,"tag":211,"props":2826,"children":2827},{"style":416},[2828],{"type":47,"value":2544},{"type":41,"tag":211,"props":2830,"children":2831},{"style":1157},[2832],{"type":47,"value":2833}," GPU Memory Peak ",{"type":41,"tag":211,"props":2835,"children":2836},{"style":416},[2837],{"type":47,"value":2544},{"type":41,"tag":211,"props":2839,"children":2840},{"style":1157},[2841],{"type":47,"value":2842}," XX% ",{"type":41,"tag":211,"props":2844,"children":2845},{"style":416},[2846],{"type":47,"value":2563},{"type":41,"tag":211,"props":2848,"children":2849},{"class":213,"line":1993},[2850],{"type":41,"tag":211,"props":2851,"children":2852},{"emptyLinePlaceholder":672},[2853],{"type":47,"value":675},{"type":41,"tag":211,"props":2855,"children":2856},{"class":213,"line":2002},[2857,2861],{"type":41,"tag":211,"props":2858,"children":2859},{"style":416},[2860],{"type":47,"value":2531},{"type":41,"tag":211,"props":2862,"children":2863},{"style":218},[2864],{"type":47,"value":2865},"Anomalies\n",{"type":41,"tag":211,"props":2867,"children":2868},{"class":213,"line":2011},[2869,2874],{"type":41,"tag":211,"props":2870,"children":2871},{"style":416},[2872],{"type":47,"value":2873},"-",{"type":41,"tag":211,"props":2875,"children":2876},{"style":1157},[2877],{"type":47,"value":2878}," List any OOM events, loss spikes, NaN occurrences, or GPU warnings detected during training.\n",{"type":41,"tag":211,"props":2880,"children":2881},{"class":213,"line":2020},[2882,2886],{"type":41,"tag":211,"props":2883,"children":2884},{"style":416},[2885],{"type":47,"value":2873},{"type":41,"tag":211,"props":2887,"children":2888},{"style":1157},[2889],{"type":47,"value":2890}," \"None\" if no anomalies were detected.\n",{"type":41,"tag":211,"props":2892,"children":2893},{"class":213,"line":2029},[2894],{"type":41,"tag":211,"props":2895,"children":2896},{"emptyLinePlaceholder":672},[2897],{"type":47,"value":675},{"type":41,"tag":211,"props":2899,"children":2900},{"class":213,"line":2037},[2901,2905],{"type":41,"tag":211,"props":2902,"children":2903},{"style":416},[2904],{"type":47,"value":2531},{"type":41,"tag":211,"props":2906,"children":2907},{"style":218},[2908],{"type":47,"value":2909},"Artifacts\n",{"type":41,"tag":211,"props":2911,"children":2912},{"class":213,"line":2046},[2913,2917,2923,2929,2934,2939,2943,2948],{"type":41,"tag":211,"props":2914,"children":2915},{"style":416},[2916],{"type":47,"value":2873},{"type":41,"tag":211,"props":2918,"children":2920},{"style":2919},"--shiki-light:#39ADB5;--shiki-light-font-weight:bold;--shiki-default:#89DDFF;--shiki-default-font-weight:bold;--shiki-dark:#89DDFF;--shiki-dark-font-weight:bold",[2921],{"type":47,"value":2922}," **",{"type":41,"tag":211,"props":2924,"children":2926},{"style":2925},"--shiki-light:#E53935;--shiki-light-font-weight:bold;--shiki-default:#F07178;--shiki-default-font-weight:bold;--shiki-dark:#F07178;--shiki-dark-font-weight:bold",[2927],{"type":47,"value":2928},"Inference-ready checkpoint:",{"type":41,"tag":211,"props":2930,"children":2931},{"style":2919},[2932],{"type":47,"value":2933},"**",{"type":41,"tag":211,"props":2935,"children":2936},{"style":416},[2937],{"type":47,"value":2938}," `",{"type":41,"tag":211,"props":2940,"children":2941},{"style":224},[2942],{"type":47,"value":984},{"type":41,"tag":211,"props":2944,"children":2945},{"style":416},[2946],{"type":47,"value":2947},"`",{"type":41,"tag":211,"props":2949,"children":2950},{"style":1157},[2951],{"type":47,"value":2952}," — always, regardless of training mode (Phase 3b merges LoRA into the base in place; full fine-tune writes here directly).\n",{"type":41,"tag":211,"props":2954,"children":2955},{"class":213,"line":2055},[2956,2960,2964,2969,2973,2977,2982,2986],{"type":41,"tag":211,"props":2957,"children":2958},{"style":416},[2959],{"type":47,"value":2873},{"type":41,"tag":211,"props":2961,"children":2962},{"style":2919},[2963],{"type":47,"value":2922},{"type":41,"tag":211,"props":2965,"children":2966},{"style":2925},[2967],{"type":47,"value":2968},"LoRA adapter backup (LoRA runs only):",{"type":41,"tag":211,"props":2970,"children":2971},{"style":2919},[2972],{"type":47,"value":2933},{"type":41,"tag":211,"props":2974,"children":2975},{"style":416},[2976],{"type":47,"value":2938},{"type":41,"tag":211,"props":2978,"children":2979},{"style":224},[2980],{"type":47,"value":2981},"assets\u002Fresults\u002F\u003Cjob_id>\u002F\u003Ctimestamp>\u002Fsafetensors\u002Fstep_\u003CN>_lora_adapter\u002F",{"type":41,"tag":211,"props":2983,"children":2984},{"style":416},[2985],{"type":47,"value":2947},{"type":41,"tag":211,"props":2987,"children":2988},{"style":1157},[2989],{"type":47,"value":2990}," — the unmerged adapter, kept for reproducibility\u002Faudit.\n",{"type":41,"tag":211,"props":2992,"children":2993},{"class":213,"line":2064},[2994,2998,3002,3007,3011,3015,3019],{"type":41,"tag":211,"props":2995,"children":2996},{"style":416},[2997],{"type":47,"value":2873},{"type":41,"tag":211,"props":2999,"children":3000},{"style":2919},[3001],{"type":47,"value":2922},{"type":41,"tag":211,"props":3003,"children":3004},{"style":2925},[3005],{"type":47,"value":3006},"Training log:",{"type":41,"tag":211,"props":3008,"children":3009},{"style":2919},[3010],{"type":47,"value":2933},{"type":41,"tag":211,"props":3012,"children":3013},{"style":416},[3014],{"type":47,"value":2938},{"type":41,"tag":211,"props":3016,"children":3017},{"style":224},[3018],{"type":47,"value":66},{"type":41,"tag":211,"props":3020,"children":3021},{"style":416},[3022],{"type":47,"value":3023},"`\n",{"type":41,"tag":891,"props":3025,"children":3027},{"id":3026},"_3d-output-summary",[3028],{"type":47,"value":3029},"3d: Output Summary",{"type":41,"tag":50,"props":3031,"children":3032},{},[3033],{"type":47,"value":3034},"Print the training report content to the user as well. If training failed, include the error details.",{"type":41,"tag":3036,"props":3037,"children":3038},"style",{},[3039],{"type":47,"value":3040},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3042,"total":1463},[3043,3061,3074,3081,3099,3111,3122],{"slug":3044,"name":3044,"fn":3045,"description":3046,"org":3047,"tags":3048,"stars":23,"repoUrl":24,"updatedAt":3060},"sop-build","orchestrate end-to-end SOP monitoring pipelines","Orchestrate the end-to-end SOP pipeline, including preflight prerequisite checks, verifying models and downloading assets, generating the DeepStream SOP microservice with RTSP output, evaluating the microservice, and building, deploying, and testing the VSS SOP blueprint. Use when asked to run the full SOP pipeline, set up the SOP pipeline from scratch, execute preflight checks, verify models, download assets, generate the SOP microservice, evaluate the microservice, build the VSS blueprint, deploy the VSS blueprint, test the VSS blueprint, or manage the complete build-evaluate-deploy-test cycle.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3049,3052,3055,3056,3057],{"name":3050,"slug":3051,"type":15},"Automation","automation",{"name":3053,"slug":3054,"type":15},"Engineering","engineering",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":3058,"slug":3059,"type":15},"Video","video","2026-07-14T05:32:48.503678",{"slug":3062,"name":3062,"fn":3063,"description":3064,"org":3065,"tags":3066,"stars":23,"repoUrl":24,"updatedAt":3073},"sop-by-action-eval","run by-action VLM evaluations","Use when running by-action VLM evaluation (per-action-clip inference + accuracy metrics) against the BP evaluation-ms HTTP API. Invoked as \u002Fsop-by-action-eval \u003Cinputs.yaml> [natural language parameter overrides]",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3067,3070,3071,3072],{"name":3068,"slug":3069,"type":15},"Evals","evals",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:13.260585",{"slug":4,"name":4,"fn":5,"description":6,"org":3075,"tags":3076,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3077,3078,3079,3080],{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"slug":3082,"name":3082,"fn":3083,"description":3084,"org":3085,"tags":3086,"stars":23,"repoUrl":24,"updatedAt":3098},"sop-data-augmentation","augment annotated datasets","Use when the user wants to run data augmentation on an annotated dataset, configure augmentation parameters, check augmentation status, or understand what each QA augmentation type does (BCQ, MCQ, GQA, DMCQ, DSQA, ENQA)",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3087,3090,3093,3094,3095],{"name":3088,"slug":3089,"type":15},"Data Engineering","data-engineering",{"name":3091,"slug":3092,"type":15},"Datasets","datasets",{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3096,"slug":3097,"type":15},"QA","qa","2026-07-14T05:36:20.305657",{"slug":3100,"name":3100,"fn":3101,"description":3102,"org":3103,"tags":3104,"stars":23,"repoUrl":24,"updatedAt":3110},"sop-ddm-finetuning","fine-tune DDM-Net models for SOP monitoring","Fine-tune DDM-Net temporal boundary detector for SOP monitoring. Use when you need to launch and monitor a DDM-Net training run with a given dataset ID.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3105,3108,3109],{"name":3106,"slug":3107,"type":15},"Deep Learning","deep-learning",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:16.203684",{"slug":3112,"name":3112,"fn":3113,"description":3114,"org":3115,"tags":3116,"stars":23,"repoUrl":24,"updatedAt":3121},"sop-e2e-inference","run end-to-end inference evaluations","Use when running the e2e evaluation pipeline (temporal segmentation + action recognition + accuracy) against the BP evaluation-ms HTTP API. Invoked as \u002Fsop-e2e-inference \u003Cinputs.yaml> [natural language parameter overrides]",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3117,3118,3119,3120],{"name":3068,"slug":3069,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:15.23958",{"slug":3123,"name":3123,"fn":3124,"description":3125,"org":3126,"tags":3127,"stars":23,"repoUrl":24,"updatedAt":3134},"sop-ft-orchestrate","orchestrate SOP fine-tuning pipelines","Autonomous end-to-end orchestrator for SOP fine-tuning. Runs the full Import → Augment → DDM Train → VLM Train → Evaluate → RCA loop. Interprets RCA findings across DDM, VLM and augment axes, applies config fixes autonomously, and iterates until success criteria are met or max_pipeline_iterations reached. Call with a path to an inputs.yaml or with natural language.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3128,3129,3130,3131],{"name":3050,"slug":3051,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3132,"slug":3133,"type":15},"Orchestration","orchestration","2026-07-14T05:36:28.162686",{"items":3136,"total":3289},[3137,3155,3173,3184,3196,3210,3223,3235,3246,3257,3271,3280],{"slug":3138,"name":3138,"fn":3139,"description":3140,"org":3141,"tags":3142,"stars":3152,"repoUrl":3153,"updatedAt":3154},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3143,3146,3149],{"name":3144,"slug":3145,"type":15},"Documentation","documentation",{"name":3147,"slug":3148,"type":15},"MCP","mcp",{"name":3150,"slug":3151,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":3156,"name":3156,"fn":3157,"description":3158,"org":3159,"tags":3160,"stars":3170,"repoUrl":3171,"updatedAt":3172},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3161,3164,3167],{"name":3162,"slug":3163,"type":15},"Containers","containers",{"name":3165,"slug":3166,"type":15},"Deployment","deployment",{"name":3168,"slug":3169,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":3174,"name":3174,"fn":3175,"description":3176,"org":3177,"tags":3178,"stars":3170,"repoUrl":3171,"updatedAt":3183},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3179,3182],{"name":3180,"slug":3181,"type":15},"CI\u002FCD","ci-cd",{"name":3165,"slug":3166,"type":15},"2026-07-14T05:25:59.97109",{"slug":3185,"name":3185,"fn":3186,"description":3187,"org":3188,"tags":3189,"stars":3170,"repoUrl":3171,"updatedAt":3195},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3190,3191,3192],{"name":3180,"slug":3181,"type":15},{"name":3165,"slug":3166,"type":15},{"name":3193,"slug":3194,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":3197,"name":3197,"fn":3198,"description":3199,"org":3200,"tags":3201,"stars":3170,"repoUrl":3171,"updatedAt":3209},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3202,3205,3206],{"name":3203,"slug":3204,"type":15},"Debugging","debugging",{"name":3193,"slug":3194,"type":15},{"name":3207,"slug":3208,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":3211,"name":3211,"fn":3212,"description":3213,"org":3214,"tags":3215,"stars":3170,"repoUrl":3171,"updatedAt":3222},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3216,3219],{"name":3217,"slug":3218,"type":15},"Best Practices","best-practices",{"name":3220,"slug":3221,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":3224,"name":3224,"fn":3225,"description":3226,"org":3227,"tags":3228,"stars":3170,"repoUrl":3171,"updatedAt":3234},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3229,3230,3233],{"name":17,"slug":18,"type":15},{"name":3231,"slug":3232,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":3236,"name":3236,"fn":3237,"description":3238,"org":3239,"tags":3240,"stars":3170,"repoUrl":3171,"updatedAt":3245},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3241,3242],{"name":3096,"slug":3097,"type":15},{"name":3243,"slug":3244,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":3247,"name":3247,"fn":3248,"description":3249,"org":3250,"tags":3251,"stars":3170,"repoUrl":3171,"updatedAt":3256},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3252,3253],{"name":3165,"slug":3166,"type":15},{"name":3254,"slug":3255,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":3258,"name":3258,"fn":3259,"description":3260,"org":3261,"tags":3262,"stars":3170,"repoUrl":3171,"updatedAt":3270},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3263,3266,3267],{"name":3264,"slug":3265,"type":15},"Code Review","code-review",{"name":3193,"slug":3194,"type":15},{"name":3268,"slug":3269,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":3272,"name":3272,"fn":3273,"description":3274,"org":3275,"tags":3276,"stars":3170,"repoUrl":3171,"updatedAt":3279},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3277,3278],{"name":3096,"slug":3097,"type":15},{"name":3243,"slug":3244,"type":15},"2026-07-14T05:25:54.928983",{"slug":3281,"name":3281,"fn":3282,"description":3283,"org":3284,"tags":3285,"stars":3170,"repoUrl":3171,"updatedAt":3288},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3286,3287],{"name":3050,"slug":3051,"type":15},{"name":3180,"slug":3181,"type":15},"2026-07-30T05:29:03.275638",496]