[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-tao-generate-video-reasoning-annotations":3,"mdc--f02eri-key":34,"related-org-nvidia-tao-generate-video-reasoning-annotations":2177,"related-repo-nvidia-tao-generate-video-reasoning-annotations":2336},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"tao-generate-video-reasoning-annotations","generate video reasoning training annotations","Multi-step video annotation pipeline that turns raw videos into Chain-of-Thought training data — multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning traces, via VLM\u002FLLM distillation. Use when the user wants to \"create video training data\", \"generate video QA datasets\", \"build CoT reasoning traces from videos\", \"auto-label videos\", or run the video_reasoning_annotation pipeline. Triggers include \"video annotation\", \"video CoT\", \"video QA\", \"chain-of-thought\", \"video captioning pipeline\", \"video distillation\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Datasets","datasets","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Video","video",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:32.626099","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Ftao-generate-video-reasoning-annotations","---\nname: tao-generate-video-reasoning-annotations\ndescription: >-\n  Multi-step video annotation pipeline that turns raw videos into\n  Chain-of-Thought training data — multi-level captions, structured\n  descriptions, and QA pairs (MCQ, binary, open-ended) with reasoning\n  traces, via VLM\u002FLLM distillation. Use when the user wants to \"create\n  video training data\", \"generate video QA datasets\", \"build CoT\n  reasoning traces from videos\", \"auto-label videos\", or run the\n  video_reasoning_annotation pipeline. Triggers include \"video\n  annotation\", \"video CoT\", \"video QA\", \"chain-of-thought\",\n  \"video captioning pipeline\", \"video distillation\".\nlicense: Apache-2.0\ncompatibility: Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash Write\ntags:\n  - video\n  - annotation\n  - chain-of-thought\n  - captioning\n  - qa-generation\n  - vlm\n  - llm\n  - auto-label\n---\n\n# Video Reasoning Annotation Pipeline\n\nGenerate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain.\n\n## Purpose\n\nTransform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as \"teacher\" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls).\n\n## Pipeline architecture\n\n```\nStep 0:  [Optional] Filter & classify videos  → Keep domain-relevant, classify anomaly vs normal\nStep 1a: Global + dense captions               → VLM: narrative summary + timestamped events\nStep 1b: Chunk captions                         → VLM: fixed-duration segment micro-captions\nStep 1c: [Optional, anomaly only] Highlight     → LLM extracts anomaly timestamp, VLM captions clip\nStep 2:  Description synthesis                  → LLM: synthesize captions into structured narrative\nStep 3:  QA generation                          → LLM: MCQ, binary, open-ended with reasoning\nStep 4:  Parse outputs                          → Per-task `tao-vl-reason-v1.0` JSON files\n```\n\nSteps are individually selectable via `workflow.steps`. The pipeline has built-in resume — each step skips already-processed videos, so re-running after a prompt tweak is safe.\n\n## Initial consultation\n\nWhen the user invokes this skill, walk through these questions in order. Don't skip — getting domain and VLM access right up front prevents wasted runs.\n\n### 1. Videos\n\n- Path to the video directory and\u002For a JSONL with `{\"video_path\": \"...\"}` per line.\n- Confirm format (`.mp4` preferred; `.avi`, `.mov`, `.mkv` also walked).\n\n### 2. Domain — drives prompt selection\n\nAsk the user: *\"What domain are these videos from?\"* Choose one of the following branches:\n\n| Domain | What to do |\n|---|---|\n| **general** | Use the default prompts. Set `prompts_module: \"\"` (or omit). The built-in `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts` covers domain-agnostic content. |\n| **traffic** (CCTV intersections, highways; dashcam excluded) | Use the reference module. Set `prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_traffic\"`, **or** copy `references\u002Fprompts_traffic.py` into the user's project and tune for their specific camera angles, then point `prompts_module` at the copy. |\n| **warehouse** (industrial site CCTV — safety, operations, security) | Same pattern. Set `prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_warehouse\"`, or copy `references\u002Fprompts_warehouse.py` and tune. |\n| **custom** (any other domain) | **Run the workshop in [references\u002Fdomain_adaptation.md](references\u002Fdomain_adaptation.md)**. It walks through: Phase 1 — question types the user wants the model to answer; Phase 2 — caption-requirements checklist; Phase 3 — fill the `[PLACEHOLDER]` markers in `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template`. The two reference modules above are working examples to model after. Do this **before** any pipeline runs. |\n\n### 3. Anomaly \u002F normal \u002F mixed\n\n- Mixed dataset → `workflow.mode: \"auto\"` (Step 0 classifies each video).\n- Pre-split anomaly only → `workflow.mode: \"anomaly\"`, drop Step 0.\n- Pre-split normal only → `workflow.mode: \"normal\"`, drop Steps 0 and 1c.\n\n### 4. VLM \u002F LLM endpoint — confirm access **before** running\n\n- **Gemini** (default for both `vlm.backend` and `llm.backend`): user needs `GOOGLE_API_KEY` set, or to put the key in the YAML.\n- **OpenAI-compatible** (Qwen via vLLM, NIM endpoint, etc.): user provides `base_url`, `model_name`, and `api_key`.\n- Steps 2–3 are text-only — a smaller\u002Fcheaper LLM is fine for `llm.backend` even when `vlm.backend` is a frontier video model.\n\nIf the user has **no endpoint at all** and wants to self-host, point them at the `skills\u002Fapplications\u002Ftao-run-inference-service` skill — a workflow that stands up a network-specific TAO inference microservice locally and exposes an OpenAI-compatible endpoint. Should support Cosmos, Qwen, and Gemma. Check `skills\u002Fapplications\u002Ftao-run-inference-service\u002Freferences\u002Fservice.yaml` for the current `valid_network_arch_config_basenames` list before relying on a specific model.\n\nIf the user doesn't have endpoint access ready and isn't ready to set one up, stop here and help them figure it out first.\n\n### 5. Pilot vs full run\n\n- **Recommend a 5–10 video pilot** when domain is `custom`, when any prompt was edited, or when this is the user's first run.\n- **Full-run is fine** for `general` \u002F `traffic` \u002F `warehouse` once the user has previously verified output quality on the same data type.\n- The pipeline has built-in resume, so a pilot followed by a full run does not re-process the pilot videos.\n\n## Quick start\n\nThe pipeline runs inside the TAO Toolkit container via the `auto_label` CLI:\n\n```bash\nauto_label generate -e \u002Fpath\u002Fto\u002Fspec.yaml \\\n    results_dir=\u002Fresults \\\n    video_reasoning_annotation.data.video_root=\u002Fvideos \\\n    video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY \\\n    video_reasoning_annotation.workflow.mode=auto\n```\n\nGenerate a default spec to start from:\n\n```bash\nauto_label default_specs results_dir=\u002Fresults module_name=auto_label\n# then set:  autolabel_type: \"video_reasoning_annotation\"\n```\n\nAll fields support Hydra dot-notation overrides on the command line. For the full YAML reference (every field, model\u002Fendpoint setup, error patterns), see [references\u002Fconfiguration.md](references\u002Fconfiguration.md).\n\n## Pilot workflow\n\nUse this when running a 5–10 video pilot:\n\n1. Run the pipeline on the pilot subset with the chosen `prompts_module` and `workflow.mode`.\n2. Inspect `results_dir\u002Fstep_1a_caption\u002Fcaptions.jsonl` — captions accurate, capturing the right level of detail?\n3. Inspect `results_dir\u002Fstep_3_qa\u002Fqa_output.jsonl` — questions meaningful, answers correct, reasoning logical?\n4. If quality is insufficient: adjust the prompts (in `prompts_module` if domain-customized, or fall back to `general` if a domain module is over-tuned), and re-run. The pipeline auto-skips already-processed videos.\n5. Once satisfied, scale to the full dataset by pointing `data.video_root` (or `data.input_jsonl_files`) at the full set and re-running with the same `results_dir` (resume) or a fresh one (full re-run).\n\nQuality compounds downstream — bad captions produce bad descriptions which produce bad QA. Focus iteration on Step 1a\u002F1b output first; descriptions and QA usually improve once captions are right.\n\n## Configuration summary\n\nKey fields (full reference in [references\u002Fconfiguration.md](references\u002Fconfiguration.md)):\n\n| Field | Default | Description |\n|---|---|---|\n| `workflow.steps` | `[\"0\",\"1a\",\"1b\",\"1c\",\"2\",\"3\",\"4\"]` | Which pipeline steps to execute |\n| `workflow.mode` | `\"auto\"` | `\"auto\"`, `\"anomaly\"`, or `\"normal\"` |\n| `vlm.backend` | `\"gemini\"` | `\"gemini\"` or `\"openai\"` (OpenAI-compatible) |\n| `llm.backend` | `\"gemini\"` | Same options; text-only, cheaper model works |\n| `workflow.max_workers` | `4` | Parallel threads per step (watch API rate limits) |\n| `license` | `\"\"` | Optional: written to `metadata.license` in step 4 outputs (e.g. `\"CC-BY-4.0\"`) |\n| `description_extra` | `\"\"` | Optional: extra text appended to per-task descriptions in step 4 metadata |\n| `prompts_module` | `\"\"` | Dotted import path to custom prompts module |\n\n## Prompts\n\n- **Built-in (general)**: `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts` — domain-agnostic, used by default.\n- **Template**: `nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template` — same 26 keys with `[PLACEHOLDER]` markers for domain customization.\n- **Reference modules** (working examples for the consultation's `traffic` \u002F `warehouse` branches): [references\u002Fprompts_traffic.py](references\u002Fprompts_traffic.py), [references\u002Fprompts_warehouse.py](references\u002Fprompts_warehouse.py).\n- **Custom domains**: see [references\u002Fdomain_adaptation.md](references\u002Fdomain_adaptation.md) for the full workshop and placeholder reference.\n\n## Inputs\n\n- **`video_root`**: Directory of videos (walked recursively for `.mp4`, `.avi`, `.mov`, `.mkv`).\n- **`input_jsonl_files`**: List of JSONL files with `{\"video_path\": \"...\"}` per line. The `video` key is also accepted; extra fields are allowed.\n- **`filter_field`**: Optional boolean field to filter JSONL entries.\n\nProvide `video_root`, `input_jsonl_files`, or both (lists merge).\n\n## Outputs\n\nAll outputs go to `results_dir\u002F` with per-step subdirectories (`step_0_filter\u002F`, `step_1a_caption\u002F`, …, `step_4_output\u002F`):\n\n- **Steps 0–3**: JSONL — one JSON object per video per line.\n- **Step 4**: One `\u003Ctask>.json` per non-empty task type, in the **`tao-vl-reason-v1.0`** envelope. Up to 10 files: `mcq.json`, `mcq_openended.json`, `bcq.json`, `bcq_openended.json`, `open_qa.json`, `causal_linkage.json`, `temporal_localization.json`, `temporal_description.json`, `scene_description.json`, `video_summarization.json`.\n\nEach step 4 file looks like:\n\n```json\n{\n  \"format\": \"tao-vl-reason-v1.0\",\n  \"metadata\": {\"type\": \"annotation\", \"task\": \"\u003Ctask>\", \"date\": \"YYYY-MM-DD\",\n               \"description\": \"\u003Cper-task + description_extra>\", \"license\": \"\u003Cfrom config>\"},\n  \"media_root\": \"\u003Cdata.video_root>\" | null,\n  \"items\": [{\"video_id\": \"...\", \"question\": \"...\", \"answer\": \"...\", \"reasoning\": \"...\"}, ...]\n}\n```\n\n`media_root` mirrors `data.video_root` (or `null` when unset); each item's `video_id` is the entry's video path with the `video_root` prefix stripped. Set `license` and `description_extra` in the spec to populate the metadata.\n\n## Prerequisites\n\n- **Container**: `tao_toolkit.pyt` (resolves to `nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt` via `versions.yaml`).\n- **ffmpeg \u002F ffprobe**: required for chunk captioning (Step 1b) and highlight extraction (Step 1c).\n- **VLM endpoint**: at least one — Gemini API key or OpenAI-compatible endpoint.\n",{"data":35,"body":49},{"name":4,"description":6,"license":26,"compatibility":36,"metadata":37,"allowed-tools":40,"tags":41},"Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).",{"author":38,"version":39},"NVIDIA Corporation","0.1.0","Read Bash Write",[22,42,43,44,45,46,47,48],"annotation","chain-of-thought","captioning","qa-generation","vlm","llm","auto-label",{"type":50,"children":51},"root",[52,61,67,74,79,85,98,111,117,122,129,183,189,202,397,403,445,457,546,582,587,593,648,654,667,758,763,801,812,818,823,918,923,929,940,1214,1220,1312,1318,1402,1420,1426,1461,1573,1578,2059,2110,2116,2171],{"type":53,"tag":54,"props":55,"children":57},"element","h1",{"id":56},"video-reasoning-annotation-pipeline",[58],{"type":59,"value":60},"text","Video Reasoning Annotation Pipeline",{"type":53,"tag":62,"props":63,"children":64},"p",{},[65],{"type":59,"value":66},"Generate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain.",{"type":53,"tag":68,"props":69,"children":71},"h2",{"id":70},"purpose",[72],{"type":59,"value":73},"Purpose",{"type":53,"tag":62,"props":75,"children":76},{},[77],{"type":59,"value":78},"Transform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as \"teacher\" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls).",{"type":53,"tag":68,"props":80,"children":82},{"id":81},"pipeline-architecture",[83],{"type":59,"value":84},"Pipeline architecture",{"type":53,"tag":86,"props":87,"children":91},"pre",{"className":88,"code":90,"language":59},[89],"language-text","Step 0:  [Optional] Filter & classify videos  → Keep domain-relevant, classify anomaly vs normal\nStep 1a: Global + dense captions               → VLM: narrative summary + timestamped events\nStep 1b: Chunk captions                         → VLM: fixed-duration segment micro-captions\nStep 1c: [Optional, anomaly only] Highlight     → LLM extracts anomaly timestamp, VLM captions clip\nStep 2:  Description synthesis                  → LLM: synthesize captions into structured narrative\nStep 3:  QA generation                          → LLM: MCQ, binary, open-ended with reasoning\nStep 4:  Parse outputs                          → Per-task `tao-vl-reason-v1.0` JSON files\n",[92],{"type":53,"tag":93,"props":94,"children":96},"code",{"__ignoreMap":95},"",[97],{"type":59,"value":90},{"type":53,"tag":62,"props":99,"children":100},{},[101,103,109],{"type":59,"value":102},"Steps are individually selectable via ",{"type":53,"tag":93,"props":104,"children":106},{"className":105},[],[107],{"type":59,"value":108},"workflow.steps",{"type":59,"value":110},". The pipeline has built-in resume — each step skips already-processed videos, so re-running after a prompt tweak is safe.",{"type":53,"tag":68,"props":112,"children":114},{"id":113},"initial-consultation",[115],{"type":59,"value":116},"Initial consultation",{"type":53,"tag":62,"props":118,"children":119},{},[120],{"type":59,"value":121},"When the user invokes this skill, walk through these questions in order. Don't skip — getting domain and VLM access right up front prevents wasted runs.",{"type":53,"tag":123,"props":124,"children":126},"h3",{"id":125},"_1-videos",[127],{"type":59,"value":128},"1. Videos",{"type":53,"tag":130,"props":131,"children":132},"ul",{},[133,147],{"type":53,"tag":134,"props":135,"children":136},"li",{},[137,139,145],{"type":59,"value":138},"Path to the video directory and\u002For a JSONL with ",{"type":53,"tag":93,"props":140,"children":142},{"className":141},[],[143],{"type":59,"value":144},"{\"video_path\": \"...\"}",{"type":59,"value":146}," per line.",{"type":53,"tag":134,"props":148,"children":149},{},[150,152,158,160,166,168,174,175,181],{"type":59,"value":151},"Confirm format (",{"type":53,"tag":93,"props":153,"children":155},{"className":154},[],[156],{"type":59,"value":157},".mp4",{"type":59,"value":159}," preferred; ",{"type":53,"tag":93,"props":161,"children":163},{"className":162},[],[164],{"type":59,"value":165},".avi",{"type":59,"value":167},", ",{"type":53,"tag":93,"props":169,"children":171},{"className":170},[],[172],{"type":59,"value":173},".mov",{"type":59,"value":167},{"type":53,"tag":93,"props":176,"children":178},{"className":177},[],[179],{"type":59,"value":180},".mkv",{"type":59,"value":182}," also walked).",{"type":53,"tag":123,"props":184,"children":186},{"id":185},"_2-domain-drives-prompt-selection",[187],{"type":59,"value":188},"2. Domain — drives prompt selection",{"type":53,"tag":62,"props":190,"children":191},{},[192,194,200],{"type":59,"value":193},"Ask the user: ",{"type":53,"tag":195,"props":196,"children":197},"em",{},[198],{"type":59,"value":199},"\"What domain are these videos from?\"",{"type":59,"value":201}," Choose one of the following branches:",{"type":53,"tag":203,"props":204,"children":205},"table",{},[206,225],{"type":53,"tag":207,"props":208,"children":209},"thead",{},[210],{"type":53,"tag":211,"props":212,"children":213},"tr",{},[214,220],{"type":53,"tag":215,"props":216,"children":217},"th",{},[218],{"type":59,"value":219},"Domain",{"type":53,"tag":215,"props":221,"children":222},{},[223],{"type":59,"value":224},"What to do",{"type":53,"tag":226,"props":227,"children":228},"tbody",{},[229,263,311,345],{"type":53,"tag":211,"props":230,"children":231},{},[232,242],{"type":53,"tag":233,"props":234,"children":235},"td",{},[236],{"type":53,"tag":237,"props":238,"children":239},"strong",{},[240],{"type":59,"value":241},"general",{"type":53,"tag":233,"props":243,"children":244},{},[245,247,253,255,261],{"type":59,"value":246},"Use the default prompts. Set ",{"type":53,"tag":93,"props":248,"children":250},{"className":249},[],[251],{"type":59,"value":252},"prompts_module: \"\"",{"type":59,"value":254}," (or omit). The built-in ",{"type":53,"tag":93,"props":256,"children":258},{"className":257},[],[259],{"type":59,"value":260},"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts",{"type":59,"value":262}," covers domain-agnostic content.",{"type":53,"tag":211,"props":264,"children":265},{},[266,276],{"type":53,"tag":233,"props":267,"children":268},{},[269,274],{"type":53,"tag":237,"props":270,"children":271},{},[272],{"type":59,"value":273},"traffic",{"type":59,"value":275}," (CCTV intersections, highways; dashcam excluded)",{"type":53,"tag":233,"props":277,"children":278},{},[279,281,287,288,293,295,301,303,309],{"type":59,"value":280},"Use the reference module. Set ",{"type":53,"tag":93,"props":282,"children":284},{"className":283},[],[285],{"type":59,"value":286},"prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_traffic\"",{"type":59,"value":167},{"type":53,"tag":237,"props":289,"children":290},{},[291],{"type":59,"value":292},"or",{"type":59,"value":294}," copy ",{"type":53,"tag":93,"props":296,"children":298},{"className":297},[],[299],{"type":59,"value":300},"references\u002Fprompts_traffic.py",{"type":59,"value":302}," into the user's project and tune for their specific camera angles, then point ",{"type":53,"tag":93,"props":304,"children":306},{"className":305},[],[307],{"type":59,"value":308},"prompts_module",{"type":59,"value":310}," at the copy.",{"type":53,"tag":211,"props":312,"children":313},{},[314,324],{"type":53,"tag":233,"props":315,"children":316},{},[317,322],{"type":53,"tag":237,"props":318,"children":319},{},[320],{"type":59,"value":321},"warehouse",{"type":59,"value":323}," (industrial site CCTV — safety, operations, security)",{"type":53,"tag":233,"props":325,"children":326},{},[327,329,335,337,343],{"type":59,"value":328},"Same pattern. Set ",{"type":53,"tag":93,"props":330,"children":332},{"className":331},[],[333],{"type":59,"value":334},"prompts_module: \"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompts_warehouse\"",{"type":59,"value":336},", or copy ",{"type":53,"tag":93,"props":338,"children":340},{"className":339},[],[341],{"type":59,"value":342},"references\u002Fprompts_warehouse.py",{"type":59,"value":344}," and tune.",{"type":53,"tag":211,"props":346,"children":347},{},[348,358],{"type":53,"tag":233,"props":349,"children":350},{},[351,356],{"type":53,"tag":237,"props":352,"children":353},{},[354],{"type":59,"value":355},"custom",{"type":59,"value":357}," (any other domain)",{"type":53,"tag":233,"props":359,"children":360},{},[361,372,374,380,382,388,390,395],{"type":53,"tag":237,"props":362,"children":363},{},[364,366],{"type":59,"value":365},"Run the workshop in ",{"type":53,"tag":367,"props":368,"children":370},"a",{"href":369},"references\u002Fdomain_adaptation.md",[371],{"type":59,"value":369},{"type":59,"value":373},". It walks through: Phase 1 — question types the user wants the model to answer; Phase 2 — caption-requirements checklist; Phase 3 — fill the ",{"type":53,"tag":93,"props":375,"children":377},{"className":376},[],[378],{"type":59,"value":379},"[PLACEHOLDER]",{"type":59,"value":381}," markers in ",{"type":53,"tag":93,"props":383,"children":385},{"className":384},[],[386],{"type":59,"value":387},"nvidia_tao_ds.auto_label.video_reasoning_annotation.prompt_template",{"type":59,"value":389},". The two reference modules above are working examples to model after. Do this ",{"type":53,"tag":237,"props":391,"children":392},{},[393],{"type":59,"value":394},"before",{"type":59,"value":396}," any pipeline runs.",{"type":53,"tag":123,"props":398,"children":400},{"id":399},"_3-anomaly-normal-mixed",[401],{"type":59,"value":402},"3. Anomaly \u002F normal \u002F mixed",{"type":53,"tag":130,"props":404,"children":405},{},[406,419,432],{"type":53,"tag":134,"props":407,"children":408},{},[409,411,417],{"type":59,"value":410},"Mixed dataset → ",{"type":53,"tag":93,"props":412,"children":414},{"className":413},[],[415],{"type":59,"value":416},"workflow.mode: \"auto\"",{"type":59,"value":418}," (Step 0 classifies each video).",{"type":53,"tag":134,"props":420,"children":421},{},[422,424,430],{"type":59,"value":423},"Pre-split anomaly only → ",{"type":53,"tag":93,"props":425,"children":427},{"className":426},[],[428],{"type":59,"value":429},"workflow.mode: \"anomaly\"",{"type":59,"value":431},", drop Step 0.",{"type":53,"tag":134,"props":433,"children":434},{},[435,437,443],{"type":59,"value":436},"Pre-split normal only → ",{"type":53,"tag":93,"props":438,"children":440},{"className":439},[],[441],{"type":59,"value":442},"workflow.mode: \"normal\"",{"type":59,"value":444},", drop Steps 0 and 1c.",{"type":53,"tag":123,"props":446,"children":448},{"id":447},"_4-vlm-llm-endpoint-confirm-access-before-running",[449,451,455],{"type":59,"value":450},"4. VLM \u002F LLM endpoint — confirm access ",{"type":53,"tag":237,"props":452,"children":453},{},[454],{"type":59,"value":394},{"type":59,"value":456}," running",{"type":53,"tag":130,"props":458,"children":459},{},[460,494,527],{"type":53,"tag":134,"props":461,"children":462},{},[463,468,470,476,478,484,486,492],{"type":53,"tag":237,"props":464,"children":465},{},[466],{"type":59,"value":467},"Gemini",{"type":59,"value":469}," (default for both ",{"type":53,"tag":93,"props":471,"children":473},{"className":472},[],[474],{"type":59,"value":475},"vlm.backend",{"type":59,"value":477}," and ",{"type":53,"tag":93,"props":479,"children":481},{"className":480},[],[482],{"type":59,"value":483},"llm.backend",{"type":59,"value":485},"): user needs ",{"type":53,"tag":93,"props":487,"children":489},{"className":488},[],[490],{"type":59,"value":491},"GOOGLE_API_KEY",{"type":59,"value":493}," set, or to put the key in the YAML.",{"type":53,"tag":134,"props":495,"children":496},{},[497,502,504,510,511,517,519,525],{"type":53,"tag":237,"props":498,"children":499},{},[500],{"type":59,"value":501},"OpenAI-compatible",{"type":59,"value":503}," (Qwen via vLLM, NIM endpoint, etc.): user provides ",{"type":53,"tag":93,"props":505,"children":507},{"className":506},[],[508],{"type":59,"value":509},"base_url",{"type":59,"value":167},{"type":53,"tag":93,"props":512,"children":514},{"className":513},[],[515],{"type":59,"value":516},"model_name",{"type":59,"value":518},", and ",{"type":53,"tag":93,"props":520,"children":522},{"className":521},[],[523],{"type":59,"value":524},"api_key",{"type":59,"value":526},".",{"type":53,"tag":134,"props":528,"children":529},{},[530,532,537,539,544],{"type":59,"value":531},"Steps 2–3 are text-only — a smaller\u002Fcheaper LLM is fine for ",{"type":53,"tag":93,"props":533,"children":535},{"className":534},[],[536],{"type":59,"value":483},{"type":59,"value":538}," even when ",{"type":53,"tag":93,"props":540,"children":542},{"className":541},[],[543],{"type":59,"value":475},{"type":59,"value":545}," is a frontier video model.",{"type":53,"tag":62,"props":547,"children":548},{},[549,551,556,558,564,566,572,574,580],{"type":59,"value":550},"If the user has ",{"type":53,"tag":237,"props":552,"children":553},{},[554],{"type":59,"value":555},"no endpoint at all",{"type":59,"value":557}," and wants to self-host, point them at the ",{"type":53,"tag":93,"props":559,"children":561},{"className":560},[],[562],{"type":59,"value":563},"skills\u002Fapplications\u002Ftao-run-inference-service",{"type":59,"value":565}," skill — a workflow that stands up a network-specific TAO inference microservice locally and exposes an OpenAI-compatible endpoint. Should support Cosmos, Qwen, and Gemma. Check ",{"type":53,"tag":93,"props":567,"children":569},{"className":568},[],[570],{"type":59,"value":571},"skills\u002Fapplications\u002Ftao-run-inference-service\u002Freferences\u002Fservice.yaml",{"type":59,"value":573}," for the current ",{"type":53,"tag":93,"props":575,"children":577},{"className":576},[],[578],{"type":59,"value":579},"valid_network_arch_config_basenames",{"type":59,"value":581}," list before relying on a specific model.",{"type":53,"tag":62,"props":583,"children":584},{},[585],{"type":59,"value":586},"If the user doesn't have endpoint access ready and isn't ready to set one up, stop here and help them figure it out first.",{"type":53,"tag":123,"props":588,"children":590},{"id":589},"_5-pilot-vs-full-run",[591],{"type":59,"value":592},"5. Pilot vs full run",{"type":53,"tag":130,"props":594,"children":595},{},[596,613,643],{"type":53,"tag":134,"props":597,"children":598},{},[599,604,606,611],{"type":53,"tag":237,"props":600,"children":601},{},[602],{"type":59,"value":603},"Recommend a 5–10 video pilot",{"type":59,"value":605}," when domain is ",{"type":53,"tag":93,"props":607,"children":609},{"className":608},[],[610],{"type":59,"value":355},{"type":59,"value":612},", when any prompt was edited, or when this is the user's first run.",{"type":53,"tag":134,"props":614,"children":615},{},[616,621,623,628,630,635,636,641],{"type":53,"tag":237,"props":617,"children":618},{},[619],{"type":59,"value":620},"Full-run is fine",{"type":59,"value":622}," for ",{"type":53,"tag":93,"props":624,"children":626},{"className":625},[],[627],{"type":59,"value":241},{"type":59,"value":629}," \u002F ",{"type":53,"tag":93,"props":631,"children":633},{"className":632},[],[634],{"type":59,"value":273},{"type":59,"value":629},{"type":53,"tag":93,"props":637,"children":639},{"className":638},[],[640],{"type":59,"value":321},{"type":59,"value":642}," once the user has previously verified output quality on the same data type.",{"type":53,"tag":134,"props":644,"children":645},{},[646],{"type":59,"value":647},"The pipeline has built-in resume, so a pilot followed by a full run does not re-process the pilot videos.",{"type":53,"tag":68,"props":649,"children":651},{"id":650},"quick-start",[652],{"type":59,"value":653},"Quick start",{"type":53,"tag":62,"props":655,"children":656},{},[657,659,665],{"type":59,"value":658},"The pipeline runs inside the TAO Toolkit container via the ",{"type":53,"tag":93,"props":660,"children":662},{"className":661},[],[663],{"type":59,"value":664},"auto_label",{"type":59,"value":666}," CLI:",{"type":53,"tag":86,"props":668,"children":672},{"className":669,"code":670,"language":671,"meta":95,"style":95},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","auto_label generate -e \u002Fpath\u002Fto\u002Fspec.yaml \\\n    results_dir=\u002Fresults \\\n    video_reasoning_annotation.data.video_root=\u002Fvideos \\\n    video_reasoning_annotation.vlm.gemini.api_key=$GOOGLE_API_KEY \\\n    video_reasoning_annotation.workflow.mode=auto\n","bash",[673],{"type":53,"tag":93,"props":674,"children":675},{"__ignoreMap":95},[676,709,722,735,749],{"type":53,"tag":677,"props":678,"children":681},"span",{"class":679,"line":680},"line",1,[682,687,693,698,703],{"type":53,"tag":677,"props":683,"children":685},{"style":684},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[686],{"type":59,"value":664},{"type":53,"tag":677,"props":688,"children":690},{"style":689},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[691],{"type":59,"value":692}," generate",{"type":53,"tag":677,"props":694,"children":695},{"style":689},[696],{"type":59,"value":697}," -e",{"type":53,"tag":677,"props":699,"children":700},{"style":689},[701],{"type":59,"value":702}," \u002Fpath\u002Fto\u002Fspec.yaml",{"type":53,"tag":677,"props":704,"children":706},{"style":705},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[707],{"type":59,"value":708}," \\\n",{"type":53,"tag":677,"props":710,"children":712},{"class":679,"line":711},2,[713,718],{"type":53,"tag":677,"props":714,"children":715},{"style":689},[716],{"type":59,"value":717},"    results_dir=\u002Fresults",{"type":53,"tag":677,"props":719,"children":720},{"style":705},[721],{"type":59,"value":708},{"type":53,"tag":677,"props":723,"children":725},{"class":679,"line":724},3,[726,731],{"type":53,"tag":677,"props":727,"children":728},{"style":689},[729],{"type":59,"value":730},"    video_reasoning_annotation.data.video_root=\u002Fvideos",{"type":53,"tag":677,"props":732,"children":733},{"style":705},[734],{"type":59,"value":708},{"type":53,"tag":677,"props":736,"children":738},{"class":679,"line":737},4,[739,744],{"type":53,"tag":677,"props":740,"children":741},{"style":689},[742],{"type":59,"value":743},"    video_reasoning_annotation.vlm.gemini.api_key=",{"type":53,"tag":677,"props":745,"children":746},{"style":705},[747],{"type":59,"value":748},"$GOOGLE_API_KEY \\\n",{"type":53,"tag":677,"props":750,"children":752},{"class":679,"line":751},5,[753],{"type":53,"tag":677,"props":754,"children":755},{"style":689},[756],{"type":59,"value":757},"    video_reasoning_annotation.workflow.mode=auto\n",{"type":53,"tag":62,"props":759,"children":760},{},[761],{"type":59,"value":762},"Generate a default spec to start from:",{"type":53,"tag":86,"props":764,"children":766},{"className":669,"code":765,"language":671,"meta":95,"style":95},"auto_label default_specs results_dir=\u002Fresults module_name=auto_label\n# then set:  autolabel_type: \"video_reasoning_annotation\"\n",[767],{"type":53,"tag":93,"props":768,"children":769},{"__ignoreMap":95},[770,792],{"type":53,"tag":677,"props":771,"children":772},{"class":679,"line":680},[773,777,782,787],{"type":53,"tag":677,"props":774,"children":775},{"style":684},[776],{"type":59,"value":664},{"type":53,"tag":677,"props":778,"children":779},{"style":689},[780],{"type":59,"value":781}," default_specs",{"type":53,"tag":677,"props":783,"children":784},{"style":689},[785],{"type":59,"value":786}," results_dir=\u002Fresults",{"type":53,"tag":677,"props":788,"children":789},{"style":689},[790],{"type":59,"value":791}," module_name=auto_label\n",{"type":53,"tag":677,"props":793,"children":794},{"class":679,"line":711},[795],{"type":53,"tag":677,"props":796,"children":798},{"style":797},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[799],{"type":59,"value":800},"# then set:  autolabel_type: \"video_reasoning_annotation\"\n",{"type":53,"tag":62,"props":802,"children":803},{},[804,806,811],{"type":59,"value":805},"All fields support Hydra dot-notation overrides on the command line. For the full YAML reference (every field, model\u002Fendpoint setup, error patterns), see ",{"type":53,"tag":367,"props":807,"children":809},{"href":808},"references\u002Fconfiguration.md",[810],{"type":59,"value":808},{"type":59,"value":526},{"type":53,"tag":68,"props":813,"children":815},{"id":814},"pilot-workflow",[816],{"type":59,"value":817},"Pilot workflow",{"type":53,"tag":62,"props":819,"children":820},{},[821],{"type":59,"value":822},"Use this when running a 5–10 video pilot:",{"type":53,"tag":824,"props":825,"children":826},"ol",{},[827,845,858,870,889],{"type":53,"tag":134,"props":828,"children":829},{},[830,832,837,838,844],{"type":59,"value":831},"Run the pipeline on the pilot subset with the chosen ",{"type":53,"tag":93,"props":833,"children":835},{"className":834},[],[836],{"type":59,"value":308},{"type":59,"value":477},{"type":53,"tag":93,"props":839,"children":841},{"className":840},[],[842],{"type":59,"value":843},"workflow.mode",{"type":59,"value":526},{"type":53,"tag":134,"props":846,"children":847},{},[848,850,856],{"type":59,"value":849},"Inspect ",{"type":53,"tag":93,"props":851,"children":853},{"className":852},[],[854],{"type":59,"value":855},"results_dir\u002Fstep_1a_caption\u002Fcaptions.jsonl",{"type":59,"value":857}," — captions accurate, capturing the right level of detail?",{"type":53,"tag":134,"props":859,"children":860},{},[861,862,868],{"type":59,"value":849},{"type":53,"tag":93,"props":863,"children":865},{"className":864},[],[866],{"type":59,"value":867},"results_dir\u002Fstep_3_qa\u002Fqa_output.jsonl",{"type":59,"value":869}," — questions meaningful, answers correct, reasoning logical?",{"type":53,"tag":134,"props":871,"children":872},{},[873,875,880,882,887],{"type":59,"value":874},"If quality is insufficient: adjust the prompts (in ",{"type":53,"tag":93,"props":876,"children":878},{"className":877},[],[879],{"type":59,"value":308},{"type":59,"value":881}," if domain-customized, or fall back to ",{"type":53,"tag":93,"props":883,"children":885},{"className":884},[],[886],{"type":59,"value":241},{"type":59,"value":888}," if a domain module is over-tuned), and re-run. The pipeline auto-skips already-processed videos.",{"type":53,"tag":134,"props":890,"children":891},{},[892,894,900,902,908,910,916],{"type":59,"value":893},"Once satisfied, scale to the full dataset by pointing ",{"type":53,"tag":93,"props":895,"children":897},{"className":896},[],[898],{"type":59,"value":899},"data.video_root",{"type":59,"value":901}," (or ",{"type":53,"tag":93,"props":903,"children":905},{"className":904},[],[906],{"type":59,"value":907},"data.input_jsonl_files",{"type":59,"value":909},") at the full set and re-running with the same ",{"type":53,"tag":93,"props":911,"children":913},{"className":912},[],[914],{"type":59,"value":915},"results_dir",{"type":59,"value":917}," (resume) or a fresh one (full re-run).",{"type":53,"tag":62,"props":919,"children":920},{},[921],{"type":59,"value":922},"Quality compounds downstream — bad captions produce bad descriptions which produce bad QA. Focus iteration on Step 1a\u002F1b output first; descriptions and QA usually improve once captions are right.",{"type":53,"tag":68,"props":924,"children":926},{"id":925},"configuration-summary",[927],{"type":59,"value":928},"Configuration summary",{"type":53,"tag":62,"props":930,"children":931},{},[932,934,938],{"type":59,"value":933},"Key fields (full reference in ",{"type":53,"tag":367,"props":935,"children":936},{"href":808},[937],{"type":59,"value":808},{"type":59,"value":939},"):",{"type":53,"tag":203,"props":941,"children":942},{},[943,964],{"type":53,"tag":207,"props":944,"children":945},{},[946],{"type":53,"tag":211,"props":947,"children":948},{},[949,954,959],{"type":53,"tag":215,"props":950,"children":951},{},[952],{"type":59,"value":953},"Field",{"type":53,"tag":215,"props":955,"children":956},{},[957],{"type":59,"value":958},"Default",{"type":53,"tag":215,"props":960,"children":961},{},[962],{"type":59,"value":963},"Description",{"type":53,"tag":226,"props":965,"children":966},{},[967,992,1035,1073,1097,1123,1165,1190],{"type":53,"tag":211,"props":968,"children":969},{},[970,978,987],{"type":53,"tag":233,"props":971,"children":972},{},[973],{"type":53,"tag":93,"props":974,"children":976},{"className":975},[],[977],{"type":59,"value":108},{"type":53,"tag":233,"props":979,"children":980},{},[981],{"type":53,"tag":93,"props":982,"children":984},{"className":983},[],[985],{"type":59,"value":986},"[\"0\",\"1a\",\"1b\",\"1c\",\"2\",\"3\",\"4\"]",{"type":53,"tag":233,"props":988,"children":989},{},[990],{"type":59,"value":991},"Which pipeline steps to execute",{"type":53,"tag":211,"props":993,"children":994},{},[995,1003,1012],{"type":53,"tag":233,"props":996,"children":997},{},[998],{"type":53,"tag":93,"props":999,"children":1001},{"className":1000},[],[1002],{"type":59,"value":843},{"type":53,"tag":233,"props":1004,"children":1005},{},[1006],{"type":53,"tag":93,"props":1007,"children":1009},{"className":1008},[],[1010],{"type":59,"value":1011},"\"auto\"",{"type":53,"tag":233,"props":1013,"children":1014},{},[1015,1020,1021,1027,1029],{"type":53,"tag":93,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":59,"value":1011},{"type":59,"value":167},{"type":53,"tag":93,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":59,"value":1026},"\"anomaly\"",{"type":59,"value":1028},", or ",{"type":53,"tag":93,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":59,"value":1034},"\"normal\"",{"type":53,"tag":211,"props":1036,"children":1037},{},[1038,1046,1055],{"type":53,"tag":233,"props":1039,"children":1040},{},[1041],{"type":53,"tag":93,"props":1042,"children":1044},{"className":1043},[],[1045],{"type":59,"value":475},{"type":53,"tag":233,"props":1047,"children":1048},{},[1049],{"type":53,"tag":93,"props":1050,"children":1052},{"className":1051},[],[1053],{"type":59,"value":1054},"\"gemini\"",{"type":53,"tag":233,"props":1056,"children":1057},{},[1058,1063,1065,1071],{"type":53,"tag":93,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":59,"value":1054},{"type":59,"value":1064}," or ",{"type":53,"tag":93,"props":1066,"children":1068},{"className":1067},[],[1069],{"type":59,"value":1070},"\"openai\"",{"type":59,"value":1072}," (OpenAI-compatible)",{"type":53,"tag":211,"props":1074,"children":1075},{},[1076,1084,1092],{"type":53,"tag":233,"props":1077,"children":1078},{},[1079],{"type":53,"tag":93,"props":1080,"children":1082},{"className":1081},[],[1083],{"type":59,"value":483},{"type":53,"tag":233,"props":1085,"children":1086},{},[1087],{"type":53,"tag":93,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":59,"value":1054},{"type":53,"tag":233,"props":1093,"children":1094},{},[1095],{"type":59,"value":1096},"Same options; text-only, cheaper model works",{"type":53,"tag":211,"props":1098,"children":1099},{},[1100,1109,1118],{"type":53,"tag":233,"props":1101,"children":1102},{},[1103],{"type":53,"tag":93,"props":1104,"children":1106},{"className":1105},[],[1107],{"type":59,"value":1108},"workflow.max_workers",{"type":53,"tag":233,"props":1110,"children":1111},{},[1112],{"type":53,"tag":93,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":59,"value":1117},"4",{"type":53,"tag":233,"props":1119,"children":1120},{},[1121],{"type":59,"value":1122},"Parallel threads per step (watch API rate limits)",{"type":53,"tag":211,"props":1124,"children":1125},{},[1126,1135,1144],{"type":53,"tag":233,"props":1127,"children":1128},{},[1129],{"type":53,"tag":93,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":59,"value":1134},"license",{"type":53,"tag":233,"props":1136,"children":1137},{},[1138],{"type":53,"tag":93,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":59,"value":1143},"\"\"",{"type":53,"tag":233,"props":1145,"children":1146},{},[1147,1149,1155,1157,1163],{"type":59,"value":1148},"Optional: written to ",{"type":53,"tag":93,"props":1150,"children":1152},{"className":1151},[],[1153],{"type":59,"value":1154},"metadata.license",{"type":59,"value":1156}," in step 4 outputs (e.g. ",{"type":53,"tag":93,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":59,"value":1162},"\"CC-BY-4.0\"",{"type":59,"value":1164},")",{"type":53,"tag":211,"props":1166,"children":1167},{},[1168,1177,1185],{"type":53,"tag":233,"props":1169,"children":1170},{},[1171],{"type":53,"tag":93,"props":1172,"children":1174},{"className":1173},[],[1175],{"type":59,"value":1176},"description_extra",{"type":53,"tag":233,"props":1178,"children":1179},{},[1180],{"type":53,"tag":93,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":59,"value":1143},{"type":53,"tag":233,"props":1186,"children":1187},{},[1188],{"type":59,"value":1189},"Optional: extra text appended to per-task descriptions in step 4 metadata",{"type":53,"tag":211,"props":1191,"children":1192},{},[1193,1201,1209],{"type":53,"tag":233,"props":1194,"children":1195},{},[1196],{"type":53,"tag":93,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":59,"value":308},{"type":53,"tag":233,"props":1202,"children":1203},{},[1204],{"type":53,"tag":93,"props":1205,"children":1207},{"className":1206},[],[1208],{"type":59,"value":1143},{"type":53,"tag":233,"props":1210,"children":1211},{},[1212],{"type":59,"value":1213},"Dotted import path to custom prompts module",{"type":53,"tag":68,"props":1215,"children":1217},{"id":1216},"prompts",[1218],{"type":59,"value":1219},"Prompts",{"type":53,"tag":130,"props":1221,"children":1222},{},[1223,1240,1263,1296],{"type":53,"tag":134,"props":1224,"children":1225},{},[1226,1231,1233,1238],{"type":53,"tag":237,"props":1227,"children":1228},{},[1229],{"type":59,"value":1230},"Built-in (general)",{"type":59,"value":1232},": ",{"type":53,"tag":93,"props":1234,"children":1236},{"className":1235},[],[1237],{"type":59,"value":260},{"type":59,"value":1239}," — domain-agnostic, used by default.",{"type":53,"tag":134,"props":1241,"children":1242},{},[1243,1248,1249,1254,1256,1261],{"type":53,"tag":237,"props":1244,"children":1245},{},[1246],{"type":59,"value":1247},"Template",{"type":59,"value":1232},{"type":53,"tag":93,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":59,"value":387},{"type":59,"value":1255}," — same 26 keys with ",{"type":53,"tag":93,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":59,"value":379},{"type":59,"value":1262}," markers for domain customization.",{"type":53,"tag":134,"props":1264,"children":1265},{},[1266,1271,1273,1278,1279,1284,1286,1290,1291,1295],{"type":53,"tag":237,"props":1267,"children":1268},{},[1269],{"type":59,"value":1270},"Reference modules",{"type":59,"value":1272}," (working examples for the consultation's ",{"type":53,"tag":93,"props":1274,"children":1276},{"className":1275},[],[1277],{"type":59,"value":273},{"type":59,"value":629},{"type":53,"tag":93,"props":1280,"children":1282},{"className":1281},[],[1283],{"type":59,"value":321},{"type":59,"value":1285}," branches): ",{"type":53,"tag":367,"props":1287,"children":1288},{"href":300},[1289],{"type":59,"value":300},{"type":59,"value":167},{"type":53,"tag":367,"props":1292,"children":1293},{"href":342},[1294],{"type":59,"value":342},{"type":59,"value":526},{"type":53,"tag":134,"props":1297,"children":1298},{},[1299,1304,1306,1310],{"type":53,"tag":237,"props":1300,"children":1301},{},[1302],{"type":59,"value":1303},"Custom domains",{"type":59,"value":1305},": see ",{"type":53,"tag":367,"props":1307,"children":1308},{"href":369},[1309],{"type":59,"value":369},{"type":59,"value":1311}," for the full workshop and placeholder reference.",{"type":53,"tag":68,"props":1313,"children":1315},{"id":1314},"inputs",[1316],{"type":59,"value":1317},"Inputs",{"type":53,"tag":130,"props":1319,"children":1320},{},[1321,1360,1388],{"type":53,"tag":134,"props":1322,"children":1323},{},[1324,1333,1335,1340,1341,1346,1347,1352,1353,1358],{"type":53,"tag":237,"props":1325,"children":1326},{},[1327],{"type":53,"tag":93,"props":1328,"children":1330},{"className":1329},[],[1331],{"type":59,"value":1332},"video_root",{"type":59,"value":1334},": Directory of videos (walked recursively for ",{"type":53,"tag":93,"props":1336,"children":1338},{"className":1337},[],[1339],{"type":59,"value":157},{"type":59,"value":167},{"type":53,"tag":93,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":59,"value":165},{"type":59,"value":167},{"type":53,"tag":93,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":59,"value":173},{"type":59,"value":167},{"type":53,"tag":93,"props":1354,"children":1356},{"className":1355},[],[1357],{"type":59,"value":180},{"type":59,"value":1359},").",{"type":53,"tag":134,"props":1361,"children":1362},{},[1363,1372,1374,1379,1381,1386],{"type":53,"tag":237,"props":1364,"children":1365},{},[1366],{"type":53,"tag":93,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":59,"value":1371},"input_jsonl_files",{"type":59,"value":1373},": List of JSONL files with ",{"type":53,"tag":93,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":59,"value":144},{"type":59,"value":1380}," per line. The ",{"type":53,"tag":93,"props":1382,"children":1384},{"className":1383},[],[1385],{"type":59,"value":22},{"type":59,"value":1387}," key is also accepted; extra fields are allowed.",{"type":53,"tag":134,"props":1389,"children":1390},{},[1391,1400],{"type":53,"tag":237,"props":1392,"children":1393},{},[1394],{"type":53,"tag":93,"props":1395,"children":1397},{"className":1396},[],[1398],{"type":59,"value":1399},"filter_field",{"type":59,"value":1401},": Optional boolean field to filter JSONL entries.",{"type":53,"tag":62,"props":1403,"children":1404},{},[1405,1407,1412,1413,1418],{"type":59,"value":1406},"Provide ",{"type":53,"tag":93,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":59,"value":1332},{"type":59,"value":167},{"type":53,"tag":93,"props":1414,"children":1416},{"className":1415},[],[1417],{"type":59,"value":1371},{"type":59,"value":1419},", or both (lists merge).",{"type":53,"tag":68,"props":1421,"children":1423},{"id":1422},"outputs",[1424],{"type":59,"value":1425},"Outputs",{"type":53,"tag":62,"props":1427,"children":1428},{},[1429,1431,1437,1439,1445,1446,1452,1454,1460],{"type":59,"value":1430},"All outputs go to ",{"type":53,"tag":93,"props":1432,"children":1434},{"className":1433},[],[1435],{"type":59,"value":1436},"results_dir\u002F",{"type":59,"value":1438}," with per-step subdirectories (",{"type":53,"tag":93,"props":1440,"children":1442},{"className":1441},[],[1443],{"type":59,"value":1444},"step_0_filter\u002F",{"type":59,"value":167},{"type":53,"tag":93,"props":1447,"children":1449},{"className":1448},[],[1450],{"type":59,"value":1451},"step_1a_caption\u002F",{"type":59,"value":1453},", …, ",{"type":53,"tag":93,"props":1455,"children":1457},{"className":1456},[],[1458],{"type":59,"value":1459},"step_4_output\u002F",{"type":59,"value":939},{"type":53,"tag":130,"props":1462,"children":1463},{},[1464,1474],{"type":53,"tag":134,"props":1465,"children":1466},{},[1467,1472],{"type":53,"tag":237,"props":1468,"children":1469},{},[1470],{"type":59,"value":1471},"Steps 0–3",{"type":59,"value":1473},": JSONL — one JSON object per video per line.",{"type":53,"tag":134,"props":1475,"children":1476},{},[1477,1482,1484,1490,1492,1501,1503,1509,1510,1516,1517,1523,1524,1530,1531,1537,1538,1544,1545,1551,1552,1558,1559,1565,1566,1572],{"type":53,"tag":237,"props":1478,"children":1479},{},[1480],{"type":59,"value":1481},"Step 4",{"type":59,"value":1483},": One ",{"type":53,"tag":93,"props":1485,"children":1487},{"className":1486},[],[1488],{"type":59,"value":1489},"\u003Ctask>.json",{"type":59,"value":1491}," per non-empty task type, in the ",{"type":53,"tag":237,"props":1493,"children":1494},{},[1495],{"type":53,"tag":93,"props":1496,"children":1498},{"className":1497},[],[1499],{"type":59,"value":1500},"tao-vl-reason-v1.0",{"type":59,"value":1502}," envelope. Up to 10 files: ",{"type":53,"tag":93,"props":1504,"children":1506},{"className":1505},[],[1507],{"type":59,"value":1508},"mcq.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1511,"children":1513},{"className":1512},[],[1514],{"type":59,"value":1515},"mcq_openended.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":59,"value":1522},"bcq.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":59,"value":1529},"bcq_openended.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1532,"children":1534},{"className":1533},[],[1535],{"type":59,"value":1536},"open_qa.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1539,"children":1541},{"className":1540},[],[1542],{"type":59,"value":1543},"causal_linkage.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1546,"children":1548},{"className":1547},[],[1549],{"type":59,"value":1550},"temporal_localization.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1553,"children":1555},{"className":1554},[],[1556],{"type":59,"value":1557},"temporal_description.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":59,"value":1564},"scene_description.json",{"type":59,"value":167},{"type":53,"tag":93,"props":1567,"children":1569},{"className":1568},[],[1570],{"type":59,"value":1571},"video_summarization.json",{"type":59,"value":526},{"type":53,"tag":62,"props":1574,"children":1575},{},[1576],{"type":59,"value":1577},"Each step 4 file looks like:",{"type":53,"tag":86,"props":1579,"children":1583},{"className":1580,"code":1581,"language":1582,"meta":95,"style":95},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"format\": \"tao-vl-reason-v1.0\",\n  \"metadata\": {\"type\": \"annotation\", \"task\": \"\u003Ctask>\", \"date\": \"YYYY-MM-DD\",\n               \"description\": \"\u003Cper-task + description_extra>\", \"license\": \"\u003Cfrom config>\"},\n  \"media_root\": \"\u003Cdata.video_root>\" | null,\n  \"items\": [{\"video_id\": \"...\", \"question\": \"...\", \"answer\": \"...\", \"reasoning\": \"...\"}, ...]\n}\n","json",[1584],{"type":53,"tag":93,"props":1585,"children":1586},{"__ignoreMap":95},[1587,1596,1638,1765,1837,1880,2050],{"type":53,"tag":677,"props":1588,"children":1589},{"class":679,"line":680},[1590],{"type":53,"tag":677,"props":1591,"children":1593},{"style":1592},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1594],{"type":59,"value":1595},"{\n",{"type":53,"tag":677,"props":1597,"children":1598},{"class":679,"line":711},[1599,1604,1610,1615,1620,1625,1629,1633],{"type":53,"tag":677,"props":1600,"children":1601},{"style":1592},[1602],{"type":59,"value":1603},"  \"",{"type":53,"tag":677,"props":1605,"children":1607},{"style":1606},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1608],{"type":59,"value":1609},"format",{"type":53,"tag":677,"props":1611,"children":1612},{"style":1592},[1613],{"type":59,"value":1614},"\"",{"type":53,"tag":677,"props":1616,"children":1617},{"style":1592},[1618],{"type":59,"value":1619},":",{"type":53,"tag":677,"props":1621,"children":1622},{"style":1592},[1623],{"type":59,"value":1624}," \"",{"type":53,"tag":677,"props":1626,"children":1627},{"style":689},[1628],{"type":59,"value":1500},{"type":53,"tag":677,"props":1630,"children":1631},{"style":1592},[1632],{"type":59,"value":1614},{"type":53,"tag":677,"props":1634,"children":1635},{"style":1592},[1636],{"type":59,"value":1637},",\n",{"type":53,"tag":677,"props":1639,"children":1640},{"class":679,"line":724},[1641,1645,1650,1654,1658,1663,1667,1672,1676,1680,1684,1688,1692,1697,1701,1706,1710,1714,1718,1723,1727,1731,1735,1740,1744,1748,1752,1757,1761],{"type":53,"tag":677,"props":1642,"children":1643},{"style":1592},[1644],{"type":59,"value":1603},{"type":53,"tag":677,"props":1646,"children":1647},{"style":1606},[1648],{"type":59,"value":1649},"metadata",{"type":53,"tag":677,"props":1651,"children":1652},{"style":1592},[1653],{"type":59,"value":1614},{"type":53,"tag":677,"props":1655,"children":1656},{"style":1592},[1657],{"type":59,"value":1619},{"type":53,"tag":677,"props":1659,"children":1660},{"style":1592},[1661],{"type":59,"value":1662}," {",{"type":53,"tag":677,"props":1664,"children":1665},{"style":1592},[1666],{"type":59,"value":1614},{"type":53,"tag":677,"props":1668,"children":1669},{"style":684},[1670],{"type":59,"value":1671},"type",{"type":53,"tag":677,"props":1673,"children":1674},{"style":1592},[1675],{"type":59,"value":1614},{"type":53,"tag":677,"props":1677,"children":1678},{"style":1592},[1679],{"type":59,"value":1619},{"type":53,"tag":677,"props":1681,"children":1682},{"style":1592},[1683],{"type":59,"value":1624},{"type":53,"tag":677,"props":1685,"children":1686},{"style":689},[1687],{"type":59,"value":42},{"type":53,"tag":677,"props":1689,"children":1690},{"style":1592},[1691],{"type":59,"value":1614},{"type":53,"tag":677,"props":1693,"children":1694},{"style":1592},[1695],{"type":59,"value":1696},",",{"type":53,"tag":677,"props":1698,"children":1699},{"style":1592},[1700],{"type":59,"value":1624},{"type":53,"tag":677,"props":1702,"children":1703},{"style":684},[1704],{"type":59,"value":1705},"task",{"type":53,"tag":677,"props":1707,"children":1708},{"style":1592},[1709],{"type":59,"value":1614},{"type":53,"tag":677,"props":1711,"children":1712},{"style":1592},[1713],{"type":59,"value":1619},{"type":53,"tag":677,"props":1715,"children":1716},{"style":1592},[1717],{"type":59,"value":1624},{"type":53,"tag":677,"props":1719,"children":1720},{"style":689},[1721],{"type":59,"value":1722},"\u003Ctask>",{"type":53,"tag":677,"props":1724,"children":1725},{"style":1592},[1726],{"type":59,"value":1614},{"type":53,"tag":677,"props":1728,"children":1729},{"style":1592},[1730],{"type":59,"value":1696},{"type":53,"tag":677,"props":1732,"children":1733},{"style":1592},[1734],{"type":59,"value":1624},{"type":53,"tag":677,"props":1736,"children":1737},{"style":684},[1738],{"type":59,"value":1739},"date",{"type":53,"tag":677,"props":1741,"children":1742},{"style":1592},[1743],{"type":59,"value":1614},{"type":53,"tag":677,"props":1745,"children":1746},{"style":1592},[1747],{"type":59,"value":1619},{"type":53,"tag":677,"props":1749,"children":1750},{"style":1592},[1751],{"type":59,"value":1624},{"type":53,"tag":677,"props":1753,"children":1754},{"style":689},[1755],{"type":59,"value":1756},"YYYY-MM-DD",{"type":53,"tag":677,"props":1758,"children":1759},{"style":1592},[1760],{"type":59,"value":1614},{"type":53,"tag":677,"props":1762,"children":1763},{"style":1592},[1764],{"type":59,"value":1637},{"type":53,"tag":677,"props":1766,"children":1767},{"class":679,"line":737},[1768,1773,1778,1782,1786,1790,1795,1799,1803,1807,1811,1815,1819,1823,1828,1832],{"type":53,"tag":677,"props":1769,"children":1770},{"style":1592},[1771],{"type":59,"value":1772},"               \"",{"type":53,"tag":677,"props":1774,"children":1775},{"style":684},[1776],{"type":59,"value":1777},"description",{"type":53,"tag":677,"props":1779,"children":1780},{"style":1592},[1781],{"type":59,"value":1614},{"type":53,"tag":677,"props":1783,"children":1784},{"style":1592},[1785],{"type":59,"value":1619},{"type":53,"tag":677,"props":1787,"children":1788},{"style":1592},[1789],{"type":59,"value":1624},{"type":53,"tag":677,"props":1791,"children":1792},{"style":689},[1793],{"type":59,"value":1794},"\u003Cper-task + description_extra>",{"type":53,"tag":677,"props":1796,"children":1797},{"style":1592},[1798],{"type":59,"value":1614},{"type":53,"tag":677,"props":1800,"children":1801},{"style":1592},[1802],{"type":59,"value":1696},{"type":53,"tag":677,"props":1804,"children":1805},{"style":1592},[1806],{"type":59,"value":1624},{"type":53,"tag":677,"props":1808,"children":1809},{"style":684},[1810],{"type":59,"value":1134},{"type":53,"tag":677,"props":1812,"children":1813},{"style":1592},[1814],{"type":59,"value":1614},{"type":53,"tag":677,"props":1816,"children":1817},{"style":1592},[1818],{"type":59,"value":1619},{"type":53,"tag":677,"props":1820,"children":1821},{"style":1592},[1822],{"type":59,"value":1624},{"type":53,"tag":677,"props":1824,"children":1825},{"style":689},[1826],{"type":59,"value":1827},"\u003Cfrom config>",{"type":53,"tag":677,"props":1829,"children":1830},{"style":1592},[1831],{"type":59,"value":1614},{"type":53,"tag":677,"props":1833,"children":1834},{"style":1592},[1835],{"type":59,"value":1836},"},\n",{"type":53,"tag":677,"props":1838,"children":1839},{"class":679,"line":751},[1840,1844,1849,1853,1857,1861,1866,1870,1875],{"type":53,"tag":677,"props":1841,"children":1842},{"style":1592},[1843],{"type":59,"value":1603},{"type":53,"tag":677,"props":1845,"children":1846},{"style":1606},[1847],{"type":59,"value":1848},"media_root",{"type":53,"tag":677,"props":1850,"children":1851},{"style":1592},[1852],{"type":59,"value":1614},{"type":53,"tag":677,"props":1854,"children":1855},{"style":1592},[1856],{"type":59,"value":1619},{"type":53,"tag":677,"props":1858,"children":1859},{"style":1592},[1860],{"type":59,"value":1624},{"type":53,"tag":677,"props":1862,"children":1863},{"style":689},[1864],{"type":59,"value":1865},"\u003Cdata.video_root>",{"type":53,"tag":677,"props":1867,"children":1868},{"style":1592},[1869],{"type":59,"value":1614},{"type":53,"tag":677,"props":1871,"children":1872},{"style":705},[1873],{"type":59,"value":1874}," | ",{"type":53,"tag":677,"props":1876,"children":1877},{"style":1592},[1878],{"type":59,"value":1879},"null,\n",{"type":53,"tag":677,"props":1881,"children":1883},{"class":679,"line":1882},6,[1884,1888,1893,1897,1901,1906,1910,1915,1919,1923,1927,1932,1936,1940,1944,1949,1953,1957,1961,1965,1969,1973,1977,1982,1986,1990,1994,1998,2002,2006,2010,2015,2019,2023,2027,2031,2035,2040,2045],{"type":53,"tag":677,"props":1885,"children":1886},{"style":1592},[1887],{"type":59,"value":1603},{"type":53,"tag":677,"props":1889,"children":1890},{"style":1606},[1891],{"type":59,"value":1892},"items",{"type":53,"tag":677,"props":1894,"children":1895},{"style":1592},[1896],{"type":59,"value":1614},{"type":53,"tag":677,"props":1898,"children":1899},{"style":1592},[1900],{"type":59,"value":1619},{"type":53,"tag":677,"props":1902,"children":1903},{"style":1592},[1904],{"type":59,"value":1905}," [{",{"type":53,"tag":677,"props":1907,"children":1908},{"style":1592},[1909],{"type":59,"value":1614},{"type":53,"tag":677,"props":1911,"children":1912},{"style":684},[1913],{"type":59,"value":1914},"video_id",{"type":53,"tag":677,"props":1916,"children":1917},{"style":1592},[1918],{"type":59,"value":1614},{"type":53,"tag":677,"props":1920,"children":1921},{"style":1592},[1922],{"type":59,"value":1619},{"type":53,"tag":677,"props":1924,"children":1925},{"style":1592},[1926],{"type":59,"value":1624},{"type":53,"tag":677,"props":1928,"children":1929},{"style":689},[1930],{"type":59,"value":1931},"...",{"type":53,"tag":677,"props":1933,"children":1934},{"style":1592},[1935],{"type":59,"value":1614},{"type":53,"tag":677,"props":1937,"children":1938},{"style":1592},[1939],{"type":59,"value":1696},{"type":53,"tag":677,"props":1941,"children":1942},{"style":1592},[1943],{"type":59,"value":1624},{"type":53,"tag":677,"props":1945,"children":1946},{"style":684},[1947],{"type":59,"value":1948},"question",{"type":53,"tag":677,"props":1950,"children":1951},{"style":1592},[1952],{"type":59,"value":1614},{"type":53,"tag":677,"props":1954,"children":1955},{"style":1592},[1956],{"type":59,"value":1619},{"type":53,"tag":677,"props":1958,"children":1959},{"style":1592},[1960],{"type":59,"value":1624},{"type":53,"tag":677,"props":1962,"children":1963},{"style":689},[1964],{"type":59,"value":1931},{"type":53,"tag":677,"props":1966,"children":1967},{"style":1592},[1968],{"type":59,"value":1614},{"type":53,"tag":677,"props":1970,"children":1971},{"style":1592},[1972],{"type":59,"value":1696},{"type":53,"tag":677,"props":1974,"children":1975},{"style":1592},[1976],{"type":59,"value":1624},{"type":53,"tag":677,"props":1978,"children":1979},{"style":684},[1980],{"type":59,"value":1981},"answer",{"type":53,"tag":677,"props":1983,"children":1984},{"style":1592},[1985],{"type":59,"value":1614},{"type":53,"tag":677,"props":1987,"children":1988},{"style":1592},[1989],{"type":59,"value":1619},{"type":53,"tag":677,"props":1991,"children":1992},{"style":1592},[1993],{"type":59,"value":1624},{"type":53,"tag":677,"props":1995,"children":1996},{"style":689},[1997],{"type":59,"value":1931},{"type":53,"tag":677,"props":1999,"children":2000},{"style":1592},[2001],{"type":59,"value":1614},{"type":53,"tag":677,"props":2003,"children":2004},{"style":1592},[2005],{"type":59,"value":1696},{"type":53,"tag":677,"props":2007,"children":2008},{"style":1592},[2009],{"type":59,"value":1624},{"type":53,"tag":677,"props":2011,"children":2012},{"style":684},[2013],{"type":59,"value":2014},"reasoning",{"type":53,"tag":677,"props":2016,"children":2017},{"style":1592},[2018],{"type":59,"value":1614},{"type":53,"tag":677,"props":2020,"children":2021},{"style":1592},[2022],{"type":59,"value":1619},{"type":53,"tag":677,"props":2024,"children":2025},{"style":1592},[2026],{"type":59,"value":1624},{"type":53,"tag":677,"props":2028,"children":2029},{"style":689},[2030],{"type":59,"value":1931},{"type":53,"tag":677,"props":2032,"children":2033},{"style":1592},[2034],{"type":59,"value":1614},{"type":53,"tag":677,"props":2036,"children":2037},{"style":1592},[2038],{"type":59,"value":2039},"},",{"type":53,"tag":677,"props":2041,"children":2042},{"style":705},[2043],{"type":59,"value":2044}," ...",{"type":53,"tag":677,"props":2046,"children":2047},{"style":1592},[2048],{"type":59,"value":2049},"]\n",{"type":53,"tag":677,"props":2051,"children":2053},{"class":679,"line":2052},7,[2054],{"type":53,"tag":677,"props":2055,"children":2056},{"style":1592},[2057],{"type":59,"value":2058},"}\n",{"type":53,"tag":62,"props":2060,"children":2061},{},[2062,2067,2069,2074,2075,2081,2083,2088,2090,2095,2097,2102,2103,2108],{"type":53,"tag":93,"props":2063,"children":2065},{"className":2064},[],[2066],{"type":59,"value":1848},{"type":59,"value":2068}," mirrors ",{"type":53,"tag":93,"props":2070,"children":2072},{"className":2071},[],[2073],{"type":59,"value":899},{"type":59,"value":901},{"type":53,"tag":93,"props":2076,"children":2078},{"className":2077},[],[2079],{"type":59,"value":2080},"null",{"type":59,"value":2082}," when unset); each item's ",{"type":53,"tag":93,"props":2084,"children":2086},{"className":2085},[],[2087],{"type":59,"value":1914},{"type":59,"value":2089}," is the entry's video path with the ",{"type":53,"tag":93,"props":2091,"children":2093},{"className":2092},[],[2094],{"type":59,"value":1332},{"type":59,"value":2096}," prefix stripped. Set ",{"type":53,"tag":93,"props":2098,"children":2100},{"className":2099},[],[2101],{"type":59,"value":1134},{"type":59,"value":477},{"type":53,"tag":93,"props":2104,"children":2106},{"className":2105},[],[2107],{"type":59,"value":1176},{"type":59,"value":2109}," in the spec to populate the metadata.",{"type":53,"tag":68,"props":2111,"children":2113},{"id":2112},"prerequisites",[2114],{"type":59,"value":2115},"Prerequisites",{"type":53,"tag":130,"props":2117,"children":2118},{},[2119,2151,2161],{"type":53,"tag":134,"props":2120,"children":2121},{},[2122,2127,2128,2134,2136,2142,2144,2150],{"type":53,"tag":237,"props":2123,"children":2124},{},[2125],{"type":59,"value":2126},"Container",{"type":59,"value":1232},{"type":53,"tag":93,"props":2129,"children":2131},{"className":2130},[],[2132],{"type":59,"value":2133},"tao_toolkit.pyt",{"type":59,"value":2135}," (resolves to ",{"type":53,"tag":93,"props":2137,"children":2139},{"className":2138},[],[2140],{"type":59,"value":2141},"nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt",{"type":59,"value":2143}," via ",{"type":53,"tag":93,"props":2145,"children":2147},{"className":2146},[],[2148],{"type":59,"value":2149},"versions.yaml",{"type":59,"value":1359},{"type":53,"tag":134,"props":2152,"children":2153},{},[2154,2159],{"type":53,"tag":237,"props":2155,"children":2156},{},[2157],{"type":59,"value":2158},"ffmpeg \u002F ffprobe",{"type":59,"value":2160},": required for chunk captioning (Step 1b) and highlight extraction (Step 1c).",{"type":53,"tag":134,"props":2162,"children":2163},{},[2164,2169],{"type":53,"tag":237,"props":2165,"children":2166},{},[2167],{"type":59,"value":2168},"VLM endpoint",{"type":59,"value":2170},": at least one — Gemini API key or OpenAI-compatible endpoint.",{"type":53,"tag":2172,"props":2173,"children":2174},"style",{},[2175],{"type":59,"value":2176},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2178,"total":2335},[2179,2197,2215,2226,2238,2252,2265,2277,2290,2301,2315,2324],{"slug":2180,"name":2180,"fn":2181,"description":2182,"org":2183,"tags":2184,"stars":2194,"repoUrl":2195,"updatedAt":2196},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2185,2188,2191],{"name":2186,"slug":2187,"type":15},"Documentation","documentation",{"name":2189,"slug":2190,"type":15},"MCP","mcp",{"name":2192,"slug":2193,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2198,"name":2198,"fn":2199,"description":2200,"org":2201,"tags":2202,"stars":2212,"repoUrl":2213,"updatedAt":2214},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2203,2206,2209],{"name":2204,"slug":2205,"type":15},"Containers","containers",{"name":2207,"slug":2208,"type":15},"Deployment","deployment",{"name":2210,"slug":2211,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2216,"name":2216,"fn":2217,"description":2218,"org":2219,"tags":2220,"stars":2212,"repoUrl":2213,"updatedAt":2225},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2221,2224],{"name":2222,"slug":2223,"type":15},"CI\u002FCD","ci-cd",{"name":2207,"slug":2208,"type":15},"2026-07-14T05:25:59.97109",{"slug":2227,"name":2227,"fn":2228,"description":2229,"org":2230,"tags":2231,"stars":2212,"repoUrl":2213,"updatedAt":2237},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2232,2233,2234],{"name":2222,"slug":2223,"type":15},{"name":2207,"slug":2208,"type":15},{"name":2235,"slug":2236,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2239,"name":2239,"fn":2240,"description":2241,"org":2242,"tags":2243,"stars":2212,"repoUrl":2213,"updatedAt":2251},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2244,2247,2248],{"name":2245,"slug":2246,"type":15},"Debugging","debugging",{"name":2235,"slug":2236,"type":15},{"name":2249,"slug":2250,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2253,"name":2253,"fn":2254,"description":2255,"org":2256,"tags":2257,"stars":2212,"repoUrl":2213,"updatedAt":2264},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2258,2261],{"name":2259,"slug":2260,"type":15},"Best Practices","best-practices",{"name":2262,"slug":2263,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2266,"name":2266,"fn":2267,"description":2268,"org":2269,"tags":2270,"stars":2212,"repoUrl":2213,"updatedAt":2276},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2271,2272,2275],{"name":17,"slug":18,"type":15},{"name":2273,"slug":2274,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2278,"name":2278,"fn":2279,"description":2280,"org":2281,"tags":2282,"stars":2212,"repoUrl":2213,"updatedAt":2289},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2283,2286],{"name":2284,"slug":2285,"type":15},"QA","qa",{"name":2287,"slug":2288,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2291,"name":2291,"fn":2292,"description":2293,"org":2294,"tags":2295,"stars":2212,"repoUrl":2213,"updatedAt":2300},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2296,2297],{"name":2207,"slug":2208,"type":15},{"name":2298,"slug":2299,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2302,"name":2302,"fn":2303,"description":2304,"org":2305,"tags":2306,"stars":2212,"repoUrl":2213,"updatedAt":2314},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2307,2310,2311],{"name":2308,"slug":2309,"type":15},"Code Review","code-review",{"name":2235,"slug":2236,"type":15},{"name":2312,"slug":2313,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2316,"name":2316,"fn":2317,"description":2318,"org":2319,"tags":2320,"stars":2212,"repoUrl":2213,"updatedAt":2323},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2321,2322],{"name":2284,"slug":2285,"type":15},{"name":2287,"slug":2288,"type":15},"2026-07-14T05:25:54.928983",{"slug":2325,"name":2325,"fn":2326,"description":2327,"org":2328,"tags":2329,"stars":2212,"repoUrl":2213,"updatedAt":2334},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2330,2333],{"name":2331,"slug":2332,"type":15},"Automation","automation",{"name":2222,"slug":2223,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2337,"total":2431},[2338,2355,2365,2379,2389,2402,2417],{"slug":2339,"name":2339,"fn":2340,"description":2341,"org":2342,"tags":2343,"stars":23,"repoUrl":24,"updatedAt":2354},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2344,2347,2350,2351],{"name":2345,"slug":2346,"type":15},"Data Analysis","data-analysis",{"name":2348,"slug":2349,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":2352,"slug":2353,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":2356,"name":2356,"fn":2357,"description":2358,"org":2359,"tags":2360,"stars":23,"repoUrl":24,"updatedAt":2364},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2361,2362,2363],{"name":2207,"slug":2208,"type":15},{"name":2298,"slug":2299,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2366,"name":2366,"fn":2367,"description":2368,"org":2369,"tags":2370,"stars":23,"repoUrl":24,"updatedAt":2378},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2371,2374,2375],{"name":2372,"slug":2373,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2376,"slug":2377,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2380,"name":2380,"fn":2381,"description":2382,"org":2383,"tags":2384,"stars":23,"repoUrl":24,"updatedAt":2388},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2385,2386,2387],{"name":2345,"slug":2346,"type":15},{"name":9,"slug":8,"type":15},{"name":2287,"slug":2288,"type":15},"2026-07-17T05:29:03.913266",{"slug":2390,"name":2390,"fn":2391,"description":2392,"org":2393,"tags":2394,"stars":23,"repoUrl":24,"updatedAt":2401},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2395,2396,2399,2400],{"name":2331,"slug":2332,"type":15},{"name":2397,"slug":2398,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"2026-07-17T05:28:53.905004",{"slug":2403,"name":2403,"fn":2404,"description":2405,"org":2406,"tags":2407,"stars":23,"repoUrl":24,"updatedAt":2416},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2408,2409,2412,2413],{"name":2207,"slug":2208,"type":15},{"name":2410,"slug":2411,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2414,"slug":2415,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2418,"name":2418,"fn":2419,"description":2420,"org":2421,"tags":2422,"stars":23,"repoUrl":24,"updatedAt":2430},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2423,2424,2427],{"name":9,"slug":8,"type":15},{"name":2425,"slug":2426,"type":15},"Quantum Computing","quantum-computing",{"name":2428,"slug":2429,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]