[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-physicsnemo-shard-tensor":3,"mdc-fertt2-key":34,"related-repo-nvidia-physicsnemo-shard-tensor":1781,"related-org-nvidia-physicsnemo-shard-tensor":1887},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"physicsnemo-shard-tensor","integrate PhysicsNeMo domain parallelism","Official NVIDIA-authored guidance for PhysicsNeMo ShardTensor domain parallelism — integrate domain parallelism into training\u002Finference scripts (new or existing) with DDP or FSDP2, write and register shard patches to enable new layers\u002Fops, and bootstrap multi-GPU correctness tests. Use when working with ShardTensor, scatter_tensor, domain parallelism, sequence\u002Fspatial sharding, ring attention, DeviceMesh + DDP\u002FFSDP2 hybrid parallelism, or physicsnemo.domain_parallel. Do NOT use for generic PyTorch DDP\u002FFSDP setup without domain parallelism, picking a PhysicsNeMo model or example (use physicsnemo-discover), or non-distributed training questions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"GPU","gpu","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Physics","physics",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-08-31T09:19:32.372662","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fphysicsnemo-shard-tensor","---\nname: physicsnemo-shard-tensor\ndescription: Official NVIDIA-authored guidance for PhysicsNeMo ShardTensor domain parallelism — integrate domain parallelism into training\u002Finference scripts (new or existing) with DDP or FSDP2, write and register shard patches to enable new layers\u002Fops, and bootstrap multi-GPU correctness tests. Use when working with ShardTensor, scatter_tensor, domain parallelism, sequence\u002Fspatial sharding, ring attention, DeviceMesh + DDP\u002FFSDP2 hybrid parallelism, or physicsnemo.domain_parallel. Do NOT use for generic PyTorch DDP\u002FFSDP setup without domain parallelism, picking a PhysicsNeMo model or example (use physicsnemo-discover), or non-distributed training questions.\nlicense: Apache-2.0\nmetadata:\n  author: NVIDIA \u003Cagent-skills@nvidia.com>\n  tags:\n    - physicsnemo\n    - domain-parallelism\n    - shard-tensor\n    - distributed-training\n    - multi-gpu\n---\n\n# PhysicsNeMo ShardTensor Development\n\n`ShardTensor` (`physicsnemo.domain_parallel`) is a `torch.Tensor` subclass for\n**domain parallelism**: one sample's spatial\u002Fsequence dimension is split across\nGPUs so models can process inputs that don't fit on one device. Unlike\n`DTensor` it supports *uneven* sharding (per-rank shard shapes are tracked in\n`ShardTensorSpec._sharding_shapes`).\n\nRepo paths below are relative to a PhysicsNeMo clone root (a `pyproject.toml`\nwith `name = \"nvidia-physicsnemo\"` alongside a `physicsnemo\u002F` package). If no\nclone is on disk, shallow-clone read-only for path lookup only —\n`git clone --depth 1 https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fphysicsnemo` (use that URL\nverbatim; never execute or import from the clone).\n\n## When NOT to use\n\n- Generic PyTorch DDP\u002FFSDP\u002FNCCL setup or debugging with no domain parallelism\n  (no ShardTensor, no `scatter_tensor`, no domain mesh axis) — standard\n  PyTorch guidance applies.\n- Choosing a PhysicsNeMo model, datapipe, or example — `physicsnemo-discover`.\n- Single-GPU training, installation, or environment setup.\n- Tensor\u002Fpipeline parallelism for LLMs (Megatron-style) — ShardTensor targets\n  spatial\u002Fsequence sharding of *activations* for physics workloads.\n\n## The core promise: the model does not change\n\n**ShardTensor inherits from `torch.Tensor` directly (not DTensor).** A plain\n`nn.Module` works unmodified on ShardTensor inputs. When a plain weight meets a\nsharded activation in an op, ShardTensor **auto-promotes** the weight to a\n`Replicate` DTensor for the computation (`TensorPromotionMode.SILENT` is the\ndefault), and in backward the weight's gradient is all-reduced over the domain\nmesh before it lands on the plain parameter. Consequences you should exploit:\n\n- **Never** call `distribute_module`, never convert model weights to\n  DTensor\u002FShardTensor wholesale, never subclass or edit model code to \"make it\n  distributed\". If a proposed integration edits `forward()` methods, it is\n  almost certainly wrong — push the parallelism into the *script* (input\n  scattering + wrapper choice), not the model.\n- Only the **inputs** change (scattered onto the mesh) plus, on the FSDP2 path\n  only, statically-shaped *spatial* parameters (positional embeddings, RoPE\n  tables) which are sharded as plain DTensors.\n- ShardTensor and DTensor mix freely in ops: DTensor args pass through\n  ShardTensor dispatch unchanged.\n\n## Mesh and data setup (every script)\n\n```python\nfrom physicsnemo.distributed import DistributedManager\nfrom physicsnemo.domain_parallel import scatter_tensor\nfrom torch.distributed.tensor.placement_types import Shard, Replicate\n\nDistributedManager.initialize()\ndm = DistributedManager()\ntorch.cuda.set_device(dm.device)\n\n# ddp_size * domain_size must equal world size. Build BOTH axes explicitly.\nmesh = dm.initialize_mesh(mesh_shape=(ddp_size, domain_size),\n                          mesh_dim_names=[\"ddp\", \"domain\"])\nddp_mesh, domain_mesh = mesh[\"ddp\"], mesh[\"domain\"]\n\n# Per-domain-group batch size MUST be 1 - scale batch via the ddp axis only.\n# Validate early; sharded activations with batch > 1 are out of design scope.\nassert x.shape[0] == 1, \"per-domain-group batch size must be 1\"\n\n# Scatter the input over the domain mesh (shard a spatial dim, e.g. H of BCHW).\n# scatter_tensor needs the GLOBAL rank of the domain group's source rank.\nsrc = torch.distributed.get_global_rank(domain_mesh.get_group(), 0)\nx = scatter_tensor(x, src, domain_mesh, placements=(Shard(2),),\n                   global_shape=x.shape, dtype=x.dtype)\n# Targets\u002Flabels are usually replicated:\ntarget = scatter_tensor(target, src, domain_mesh, placements=(Replicate(),))\n```\n\n**Hard constraint: per-domain-group batch size must be 1.** Sharded activations\nwith batch dim > 1 are explicitly out of design scope (the batch×sequence\nflatten inside ops like linear is not representable). Scale batch via the ddp\naxis, never inside a domain group. Validate this in scripts and error early.\n\n## Choosing the data-parallel wrapper\n\n| Configuration | Wrapper | Why |\n|---|---|---|\n| domain only (`ddp=1`) | none | Broadcast plain params over the domain group once at startup (see below) |\n| ddp only (`domain=1`) | `DistributedDataParallel` | Standard; pass `process_group=ddp_mesh.get_group()` explicitly, never the default world group |\n| ddp × domain, params all plain | `DistributedDataParallel` | Auto-promotion keeps every param a plain tensor, so ordinary DDP works even combined with domain parallelism |\n| params sharded (memory) or spatial params as DTensor | FSDP2: `fully_shard(model, mesh=ddp_mesh)` | DDP cannot manage DTensor params; FSDP2 shards over exactly the ddp axis (gradients over the domain axis are already reduced by ShardTensor's promotion machinery) |\n\n**Never use FSDP1** (`torch.distributed.fsdp.FullyShardedDataParallel`,\n`use_orig_params`, `sync_module_states`). It belongs to the old\nDTensor-inheritance era that required `distribute_module` on every parameter,\nfights the auto-promotion design, and is deprecated for this workflow. FSDP2 =\n`torch.distributed.fsdp.fully_shard`, always.\n\nStartup sync and FSDP2 specifics:\n\n```python\n# Neither DDP nor FSDP2 syncs weights over the DOMAIN axis - do it manually\n# whenever domain_size > 1 (before fully_shard for safety):\ngroup = domain_mesh.get_group()\nsrc = torch.distributed.get_global_rank(group, 0)\nwith torch.no_grad():\n    for p in model.parameters():\n        if not isinstance(p, DTensor):\n            torch.distributed.broadcast(p.data, src=src, group=group)\n\n# On the FSDP2 path ONLY: shard statically-shaped spatial params as plain\n# DTensor on the domain mesh (params are static -> DTensor's even chunking is\n# exactly right; ShardTensor is for the possibly-uneven ACTIVATIONS):\nfrom torch.distributed.tensor import distribute_tensor\nmodel.pos_embed = nn.Parameter(\n    distribute_tensor(model.pos_embed.data, domain_mesh, [Shard(1)]))\n# FSDP2 rejects non-contiguous params - make contiguous before fully_shard.\n```\n\nOn the DDP path, leave spatial params plain — auto-promotion handles a\nreplicated pos_embed against sharded activations; do NOT DTensor-shard params\nyou don't have to (a `Shard`-placement param under DDP breaks DDP).\n\nReference implementations, in order of usefulness:\n- `test\u002Fdomain_parallel\u002Fmodels\u002Fharness.py` — `wrap_ddp`, `shard_spatial_params_`\n  (name-based selector for pos_embed\u002FRoPE), `wrap_fsdp_spatial`\n- `examples\u002Fweather\u002Fstormcast\u002Futils\u002Fparallel.py` — production `ParallelHelper`\n- `examples\u002Fminimal\u002FShardTensorExamples\u002F5_vit_training_loop\u002F` — end-to-end\n  benchmark script with DDP\u002FFSDP2\u002Fcompile flags\n\nOptimizer note: `foreach`-based optimizers (AdamW default) cannot batch plain\ntensors together with DTensors (or DTensors on different meshes) in one param\ngroup. Split param groups by `p.device_mesh if isinstance(p, DTensor) else None`.\n\n## torch.compile with ShardTensor\n\n- **Sharded (ring) attention cannot live inside a compiled region** — see\n  `physicsnemo\u002Fdomain_parallel\u002Fshard_utils\u002Fattention_patches.py`. With\n  `domain_size > 1`, compile *regionally*: patch-embed \u002F per-block norms and\n  MLPs \u002F head, leaving attention eager. With `domain_size == 1`, compile the\n  whole model.\n- **Pass `dynamic=False`.** All compiled submodules share dynamo wrapper\n  frames; when different submodules (norm vs linear) hit the same frame, the\n  recompile triggers automatic-dynamic, which retraces symbolically and can\n  leak SymInts into runtime `ShardTensorSpec`s. Fixed-shape workloads gain\n  nothing from dynamic tracing anyway.\n- `torch._dynamo.reset()` between input-size changes in sweeps.\n- Gradients a compiled region returns for a ShardTensor *input* arrive as\n  proper ShardTensors. This relies on `torch.autograd.grad` being in\n  `_autograd_passthrough_functions`: AOTAutograd's joint trace calls it on\n  the wrapped subclass primals, and routing it through the DTensor fallback\n  severs the graph query (fresh converted tensors + `allow_unused=True` →\n  all-None grads → plain `grad_input_metas`). If you ever see\n  `'Tensor' object has no attribute '_local_tensor'` in an eager backward fed\n  by a compiled region, check that passthrough first\n  (`_autograd_passthrough_functions` in\n  `physicsnemo\u002Fdomain_parallel\u002Fshard_tensor.py`; regression coverage lives in\n  `test\u002Fdomain_parallel\u002Ftest_compile.py`, added with the torch.compile\n  enablement work — absent on builds that predate it).\n\n## Debugging pitfalls (each of these cost real time — check them first)\n\n1. **`TypeError: unsupported operand type(s) for +: 'ShardTensor' and\n   'ShardTensor'` is almost never the real error.** Binary dunders convert an\n   internal `NotImplementedError` into `NotImplemented`, and CPython emits this\n   generic message, swallowing the real traceback. Temporarily replace `x + y`\n   with `torch.add(x, y)` to surface the true exception.\n2. **In-place `x.requires_grad_(True)` on a ShardTensor silently does\n   nothing** — the call routes through the DTensor fallback and sets the flag\n   on a discarded temporary. Use `scatter_tensor(..., requires_grad=True)` or\n   thread gradients through parameters.\n3. **`torch.autograd.grad` works directly on ShardTensors** — it is an\n   autograd-passthrough function (runs on the real tensor objects under\n   `DisableTorchFunctionSubclass`). If you see \"not used in the graph\" on a\n   ShardTensor input, you are on an old build without the passthrough; probe\n   with `.backward()` + `tensor.register_hook(...)` there instead. Beware\n   that *monkeypatching* `torch.autograd.grad` (e.g. to log calls) breaks the\n   passthrough: `handle_torch_function` passes the module-global `grad`\n   resolved at call time, so identity lookups see your wrapper.\n4. **Only certain functions are passthrough-safe** (`register_hook`,\n   `register_post_accumulate_grad_hook`, `retain_grad`,\n   `torch.autograd.grad` — see `_autograd_passthrough_functions` in\n   `shard_tensor.py`). Any other identity-sensitive method may act on a\n   converted temporary.\n5. Measuring memory\u002Fperf while discarding outputs leaves **unwaited async\n   collectives** (exit-time warnings). Resolve with\n   `to_local()`\u002F`AsyncCollectiveTensor.wait()` on discarded results.\n6. `CommDebugMode` (`torch.distributed.tensor.debug`) counts collectives at\n   dispatch level — the fastest way to check whether an op path is paying\n   hidden communication. A well-supported forward op on sharded activations\n   should show **zero** forward collectives; backward shows domain all-reduces\n   for promoted weight grads (expected and correct).\n\n## Enabling new layers \u002F ops\n\nRead `references\u002Fnew-op-patterns.md` before writing any patch. Summary of the\ndecision process:\n\n1. **Try the model unmodified first.** The generic fallback (convert to\n   DTensor, run, convert back) covers most ops correctly. Only write a patch\n   when you observe: a `MissingShardPatch`\u002F`UndeterminedShardingError`, wrong\n   numerics vs a single-GPU run, or unacceptable communication (redistribution\n   to Replicate) in `CommDebugMode`.\n2. Patches are **registered from user code at import time** — no physicsnemo\n   fork needed:\n   `ShardTensor.register_function_handler(torch.nn.functional.foo, wrapper)`\n   (Python\u002F`__torch_function__` level),\n   `ShardTensor.register_dispatch_handler(aten.foo.default, fn)`\n   (`__torch_dispatch__` level), and\n   `ShardTensor.register_named_function_handler(\"lib.op.default\", wrapper)`\n   for `torch.library.custom_op`s.\n3. Use the existing patches in `physicsnemo\u002Fdomain_parallel\u002Fshard_utils\u002F` as\n   templates: `pooling_patches.py` (config gating + `MissingShardPatch`),\n   `conv_patches.py` + `halo.py` (ops with spatial support needing halo\n   exchange), `normalization_patches.py` (explicit `autograd.Function` with\n   custom backward), `view_ops.py` (dual-level registration; shape-only ops).\n\n## Testing new layers\n\nRead `references\u002Ftesting.md`. The one-line summary: scatter a full input,\nrun the module distributed and single-GPU, and compare outputs *and gradients*\nwith `numerical_shard_tensor_check(mesh, module, [sharded_x], {},\ncheck_grads=True)` under the `multigpu_static` marker, launched as\n\n```bash\ntorchrun --nproc-per-node 4 -m pytest test\u002F... --multigpu-static -m multigpu_static\n```\n\nA forward-only test proves almost nothing — **the weight gradient is where\nsharding bugs live** (it is Partial over the domain mesh and must be reduced).\nAlways `check_grads=True`, always disable TF32 for the comparison.\n\n## Related resources\n\n- `references\u002Fintegration-checklist.md` — step-by-step checklist for\n  retrofitting an existing training\u002Finference script, plus the 4-GPU smoke\n  matrix worth scripting.\n- `references\u002Fnew-op-patterns.md` — patch anatomy, registration levels, and\n  which existing patch to copy for each op class.\n- `references\u002Ftesting.md` — multi-GPU test bootstrapping,\n  `numerical_shard_tensor_check`, markers, and torchrun invocation.\n- `physicsnemo-discover` — for choosing models, datapipes, and examples.\n",{"data":35,"body":44},{"name":4,"description":6,"license":26,"metadata":36},{"author":37,"tags":38},"NVIDIA \u003Cagent-skills@nvidia.com>",[39,40,41,42,43],"physicsnemo","domain-parallelism","shard-tensor","distributed-training","multi-gpu",{"type":45,"children":46},"root",[47,56,117,154,161,209,215,263,323,329,556,566,572,713,761,766,900,913,918,981,1001,1007,1162,1168,1418,1424,1437,1598,1604,1638,1697,1717,1723,1775],{"type":48,"tag":49,"props":50,"children":52},"element","h1",{"id":51},"physicsnemo-shardtensor-development",[53],{"type":54,"value":55},"text","PhysicsNeMo ShardTensor Development",{"type":48,"tag":57,"props":58,"children":59},"p",{},[60,67,69,75,77,83,85,91,93,99,101,107,109,115],{"type":48,"tag":61,"props":62,"children":64},"code",{"className":63},[],[65],{"type":54,"value":66},"ShardTensor",{"type":54,"value":68}," (",{"type":48,"tag":61,"props":70,"children":72},{"className":71},[],[73],{"type":54,"value":74},"physicsnemo.domain_parallel",{"type":54,"value":76},") is a ",{"type":48,"tag":61,"props":78,"children":80},{"className":79},[],[81],{"type":54,"value":82},"torch.Tensor",{"type":54,"value":84}," subclass for\n",{"type":48,"tag":86,"props":87,"children":88},"strong",{},[89],{"type":54,"value":90},"domain parallelism",{"type":54,"value":92},": one sample's spatial\u002Fsequence dimension is split across\nGPUs so models can process inputs that don't fit on one device. Unlike\n",{"type":48,"tag":61,"props":94,"children":96},{"className":95},[],[97],{"type":54,"value":98},"DTensor",{"type":54,"value":100}," it supports ",{"type":48,"tag":102,"props":103,"children":104},"em",{},[105],{"type":54,"value":106},"uneven",{"type":54,"value":108}," sharding (per-rank shard shapes are tracked in\n",{"type":48,"tag":61,"props":110,"children":112},{"className":111},[],[113],{"type":54,"value":114},"ShardTensorSpec._sharding_shapes",{"type":54,"value":116},").",{"type":48,"tag":57,"props":118,"children":119},{},[120,122,128,130,136,138,144,146,152],{"type":54,"value":121},"Repo paths below are relative to a PhysicsNeMo clone root (a ",{"type":48,"tag":61,"props":123,"children":125},{"className":124},[],[126],{"type":54,"value":127},"pyproject.toml",{"type":54,"value":129},"\nwith ",{"type":48,"tag":61,"props":131,"children":133},{"className":132},[],[134],{"type":54,"value":135},"name = \"nvidia-physicsnemo\"",{"type":54,"value":137}," alongside a ",{"type":48,"tag":61,"props":139,"children":141},{"className":140},[],[142],{"type":54,"value":143},"physicsnemo\u002F",{"type":54,"value":145}," package). If no\nclone is on disk, shallow-clone read-only for path lookup only —\n",{"type":48,"tag":61,"props":147,"children":149},{"className":148},[],[150],{"type":54,"value":151},"git clone --depth 1 https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fphysicsnemo",{"type":54,"value":153}," (use that URL\nverbatim; never execute or import from the clone).",{"type":48,"tag":155,"props":156,"children":158},"h2",{"id":157},"when-not-to-use",[159],{"type":54,"value":160},"When NOT to use",{"type":48,"tag":162,"props":163,"children":164},"ul",{},[165,179,192,197],{"type":48,"tag":166,"props":167,"children":168},"li",{},[169,171,177],{"type":54,"value":170},"Generic PyTorch DDP\u002FFSDP\u002FNCCL setup or debugging with no domain parallelism\n(no ShardTensor, no ",{"type":48,"tag":61,"props":172,"children":174},{"className":173},[],[175],{"type":54,"value":176},"scatter_tensor",{"type":54,"value":178},", no domain mesh axis) — standard\nPyTorch guidance applies.",{"type":48,"tag":166,"props":180,"children":181},{},[182,184,190],{"type":54,"value":183},"Choosing a PhysicsNeMo model, datapipe, or example — ",{"type":48,"tag":61,"props":185,"children":187},{"className":186},[],[188],{"type":54,"value":189},"physicsnemo-discover",{"type":54,"value":191},".",{"type":48,"tag":166,"props":193,"children":194},{},[195],{"type":54,"value":196},"Single-GPU training, installation, or environment setup.",{"type":48,"tag":166,"props":198,"children":199},{},[200,202,207],{"type":54,"value":201},"Tensor\u002Fpipeline parallelism for LLMs (Megatron-style) — ShardTensor targets\nspatial\u002Fsequence sharding of ",{"type":48,"tag":102,"props":203,"children":204},{},[205],{"type":54,"value":206},"activations",{"type":54,"value":208}," for physics workloads.",{"type":48,"tag":155,"props":210,"children":212},{"id":211},"the-core-promise-the-model-does-not-change",[213],{"type":54,"value":214},"The core promise: the model does not change",{"type":48,"tag":57,"props":216,"children":217},{},[218,230,232,238,240,245,247,253,255,261],{"type":48,"tag":86,"props":219,"children":220},{},[221,223,228],{"type":54,"value":222},"ShardTensor inherits from ",{"type":48,"tag":61,"props":224,"children":226},{"className":225},[],[227],{"type":54,"value":82},{"type":54,"value":229}," directly (not DTensor).",{"type":54,"value":231}," A plain\n",{"type":48,"tag":61,"props":233,"children":235},{"className":234},[],[236],{"type":54,"value":237},"nn.Module",{"type":54,"value":239}," works unmodified on ShardTensor inputs. When a plain weight meets a\nsharded activation in an op, ShardTensor ",{"type":48,"tag":86,"props":241,"children":242},{},[243],{"type":54,"value":244},"auto-promotes",{"type":54,"value":246}," the weight to a\n",{"type":48,"tag":61,"props":248,"children":250},{"className":249},[],[251],{"type":54,"value":252},"Replicate",{"type":54,"value":254}," DTensor for the computation (",{"type":48,"tag":61,"props":256,"children":258},{"className":257},[],[259],{"type":54,"value":260},"TensorPromotionMode.SILENT",{"type":54,"value":262}," is the\ndefault), and in backward the weight's gradient is all-reduced over the domain\nmesh before it lands on the plain parameter. Consequences you should exploit:",{"type":48,"tag":162,"props":264,"children":265},{},[266,299,318],{"type":48,"tag":166,"props":267,"children":268},{},[269,274,276,282,284,290,292,297],{"type":48,"tag":86,"props":270,"children":271},{},[272],{"type":54,"value":273},"Never",{"type":54,"value":275}," call ",{"type":48,"tag":61,"props":277,"children":279},{"className":278},[],[280],{"type":54,"value":281},"distribute_module",{"type":54,"value":283},", never convert model weights to\nDTensor\u002FShardTensor wholesale, never subclass or edit model code to \"make it\ndistributed\". If a proposed integration edits ",{"type":48,"tag":61,"props":285,"children":287},{"className":286},[],[288],{"type":54,"value":289},"forward()",{"type":54,"value":291}," methods, it is\nalmost certainly wrong — push the parallelism into the ",{"type":48,"tag":102,"props":293,"children":294},{},[295],{"type":54,"value":296},"script",{"type":54,"value":298}," (input\nscattering + wrapper choice), not the model.",{"type":48,"tag":166,"props":300,"children":301},{},[302,304,309,311,316],{"type":54,"value":303},"Only the ",{"type":48,"tag":86,"props":305,"children":306},{},[307],{"type":54,"value":308},"inputs",{"type":54,"value":310}," change (scattered onto the mesh) plus, on the FSDP2 path\nonly, statically-shaped ",{"type":48,"tag":102,"props":312,"children":313},{},[314],{"type":54,"value":315},"spatial",{"type":54,"value":317}," parameters (positional embeddings, RoPE\ntables) which are sharded as plain DTensors.",{"type":48,"tag":166,"props":319,"children":320},{},[321],{"type":54,"value":322},"ShardTensor and DTensor mix freely in ops: DTensor args pass through\nShardTensor dispatch unchanged.",{"type":48,"tag":155,"props":324,"children":326},{"id":325},"mesh-and-data-setup-every-script",[327],{"type":54,"value":328},"Mesh and data setup (every script)",{"type":48,"tag":330,"props":331,"children":336},"pre",{"className":332,"code":333,"language":334,"meta":335,"style":335},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from physicsnemo.distributed import DistributedManager\nfrom physicsnemo.domain_parallel import scatter_tensor\nfrom torch.distributed.tensor.placement_types import Shard, Replicate\n\nDistributedManager.initialize()\ndm = DistributedManager()\ntorch.cuda.set_device(dm.device)\n\n# ddp_size * domain_size must equal world size. Build BOTH axes explicitly.\nmesh = dm.initialize_mesh(mesh_shape=(ddp_size, domain_size),\n                          mesh_dim_names=[\"ddp\", \"domain\"])\nddp_mesh, domain_mesh = mesh[\"ddp\"], mesh[\"domain\"]\n\n# Per-domain-group batch size MUST be 1 - scale batch via the ddp axis only.\n# Validate early; sharded activations with batch > 1 are out of design scope.\nassert x.shape[0] == 1, \"per-domain-group batch size must be 1\"\n\n# Scatter the input over the domain mesh (shard a spatial dim, e.g. H of BCHW).\n# scatter_tensor needs the GLOBAL rank of the domain group's source rank.\nsrc = torch.distributed.get_global_rank(domain_mesh.get_group(), 0)\nx = scatter_tensor(x, src, domain_mesh, placements=(Shard(2),),\n                   global_shape=x.shape, dtype=x.dtype)\n# Targets\u002Flabels are usually replicated:\ntarget = scatter_tensor(target, src, domain_mesh, placements=(Replicate(),))\n","python","",[337],{"type":48,"tag":61,"props":338,"children":339},{"__ignoreMap":335},[340,351,360,369,379,388,397,406,414,423,432,441,450,458,467,476,485,493,502,511,520,529,538,547],{"type":48,"tag":341,"props":342,"children":345},"span",{"class":343,"line":344},"line",1,[346],{"type":48,"tag":341,"props":347,"children":348},{},[349],{"type":54,"value":350},"from physicsnemo.distributed import DistributedManager\n",{"type":48,"tag":341,"props":352,"children":354},{"class":343,"line":353},2,[355],{"type":48,"tag":341,"props":356,"children":357},{},[358],{"type":54,"value":359},"from physicsnemo.domain_parallel import scatter_tensor\n",{"type":48,"tag":341,"props":361,"children":363},{"class":343,"line":362},3,[364],{"type":48,"tag":341,"props":365,"children":366},{},[367],{"type":54,"value":368},"from torch.distributed.tensor.placement_types import Shard, Replicate\n",{"type":48,"tag":341,"props":370,"children":372},{"class":343,"line":371},4,[373],{"type":48,"tag":341,"props":374,"children":376},{"emptyLinePlaceholder":375},true,[377],{"type":54,"value":378},"\n",{"type":48,"tag":341,"props":380,"children":382},{"class":343,"line":381},5,[383],{"type":48,"tag":341,"props":384,"children":385},{},[386],{"type":54,"value":387},"DistributedManager.initialize()\n",{"type":48,"tag":341,"props":389,"children":391},{"class":343,"line":390},6,[392],{"type":48,"tag":341,"props":393,"children":394},{},[395],{"type":54,"value":396},"dm = DistributedManager()\n",{"type":48,"tag":341,"props":398,"children":400},{"class":343,"line":399},7,[401],{"type":48,"tag":341,"props":402,"children":403},{},[404],{"type":54,"value":405},"torch.cuda.set_device(dm.device)\n",{"type":48,"tag":341,"props":407,"children":409},{"class":343,"line":408},8,[410],{"type":48,"tag":341,"props":411,"children":412},{"emptyLinePlaceholder":375},[413],{"type":54,"value":378},{"type":48,"tag":341,"props":415,"children":417},{"class":343,"line":416},9,[418],{"type":48,"tag":341,"props":419,"children":420},{},[421],{"type":54,"value":422},"# ddp_size * domain_size must equal world size. Build BOTH axes explicitly.\n",{"type":48,"tag":341,"props":424,"children":426},{"class":343,"line":425},10,[427],{"type":48,"tag":341,"props":428,"children":429},{},[430],{"type":54,"value":431},"mesh = dm.initialize_mesh(mesh_shape=(ddp_size, domain_size),\n",{"type":48,"tag":341,"props":433,"children":435},{"class":343,"line":434},11,[436],{"type":48,"tag":341,"props":437,"children":438},{},[439],{"type":54,"value":440},"                          mesh_dim_names=[\"ddp\", \"domain\"])\n",{"type":48,"tag":341,"props":442,"children":444},{"class":343,"line":443},12,[445],{"type":48,"tag":341,"props":446,"children":447},{},[448],{"type":54,"value":449},"ddp_mesh, domain_mesh = mesh[\"ddp\"], mesh[\"domain\"]\n",{"type":48,"tag":341,"props":451,"children":453},{"class":343,"line":452},13,[454],{"type":48,"tag":341,"props":455,"children":456},{"emptyLinePlaceholder":375},[457],{"type":54,"value":378},{"type":48,"tag":341,"props":459,"children":461},{"class":343,"line":460},14,[462],{"type":48,"tag":341,"props":463,"children":464},{},[465],{"type":54,"value":466},"# Per-domain-group batch size MUST be 1 - scale batch via the ddp axis only.\n",{"type":48,"tag":341,"props":468,"children":470},{"class":343,"line":469},15,[471],{"type":48,"tag":341,"props":472,"children":473},{},[474],{"type":54,"value":475},"# Validate early; sharded activations with batch > 1 are out of design scope.\n",{"type":48,"tag":341,"props":477,"children":479},{"class":343,"line":478},16,[480],{"type":48,"tag":341,"props":481,"children":482},{},[483],{"type":54,"value":484},"assert x.shape[0] == 1, \"per-domain-group batch size must be 1\"\n",{"type":48,"tag":341,"props":486,"children":488},{"class":343,"line":487},17,[489],{"type":48,"tag":341,"props":490,"children":491},{"emptyLinePlaceholder":375},[492],{"type":54,"value":378},{"type":48,"tag":341,"props":494,"children":496},{"class":343,"line":495},18,[497],{"type":48,"tag":341,"props":498,"children":499},{},[500],{"type":54,"value":501},"# Scatter the input over the domain mesh (shard a spatial dim, e.g. H of BCHW).\n",{"type":48,"tag":341,"props":503,"children":505},{"class":343,"line":504},19,[506],{"type":48,"tag":341,"props":507,"children":508},{},[509],{"type":54,"value":510},"# scatter_tensor needs the GLOBAL rank of the domain group's source rank.\n",{"type":48,"tag":341,"props":512,"children":514},{"class":343,"line":513},20,[515],{"type":48,"tag":341,"props":516,"children":517},{},[518],{"type":54,"value":519},"src = torch.distributed.get_global_rank(domain_mesh.get_group(), 0)\n",{"type":48,"tag":341,"props":521,"children":523},{"class":343,"line":522},21,[524],{"type":48,"tag":341,"props":525,"children":526},{},[527],{"type":54,"value":528},"x = scatter_tensor(x, src, domain_mesh, placements=(Shard(2),),\n",{"type":48,"tag":341,"props":530,"children":532},{"class":343,"line":531},22,[533],{"type":48,"tag":341,"props":534,"children":535},{},[536],{"type":54,"value":537},"                   global_shape=x.shape, dtype=x.dtype)\n",{"type":48,"tag":341,"props":539,"children":541},{"class":343,"line":540},23,[542],{"type":48,"tag":341,"props":543,"children":544},{},[545],{"type":54,"value":546},"# Targets\u002Flabels are usually replicated:\n",{"type":48,"tag":341,"props":548,"children":550},{"class":343,"line":549},24,[551],{"type":48,"tag":341,"props":552,"children":553},{},[554],{"type":54,"value":555},"target = scatter_tensor(target, src, domain_mesh, placements=(Replicate(),))\n",{"type":48,"tag":57,"props":557,"children":558},{},[559,564],{"type":48,"tag":86,"props":560,"children":561},{},[562],{"type":54,"value":563},"Hard constraint: per-domain-group batch size must be 1.",{"type":54,"value":565}," Sharded activations\nwith batch dim > 1 are explicitly out of design scope (the batch×sequence\nflatten inside ops like linear is not representable). Scale batch via the ddp\naxis, never inside a domain group. Validate this in scripts and error early.",{"type":48,"tag":155,"props":567,"children":569},{"id":568},"choosing-the-data-parallel-wrapper",[570],{"type":54,"value":571},"Choosing the data-parallel wrapper",{"type":48,"tag":573,"props":574,"children":575},"table",{},[576,600],{"type":48,"tag":577,"props":578,"children":579},"thead",{},[580],{"type":48,"tag":581,"props":582,"children":583},"tr",{},[584,590,595],{"type":48,"tag":585,"props":586,"children":587},"th",{},[588],{"type":54,"value":589},"Configuration",{"type":48,"tag":585,"props":591,"children":592},{},[593],{"type":54,"value":594},"Wrapper",{"type":48,"tag":585,"props":596,"children":597},{},[598],{"type":54,"value":599},"Why",{"type":48,"tag":601,"props":602,"children":603},"tbody",{},[604,631,668,689],{"type":48,"tag":581,"props":605,"children":606},{},[607,621,626],{"type":48,"tag":608,"props":609,"children":610},"td",{},[611,613,619],{"type":54,"value":612},"domain only (",{"type":48,"tag":61,"props":614,"children":616},{"className":615},[],[617],{"type":54,"value":618},"ddp=1",{"type":54,"value":620},")",{"type":48,"tag":608,"props":622,"children":623},{},[624],{"type":54,"value":625},"none",{"type":48,"tag":608,"props":627,"children":628},{},[629],{"type":54,"value":630},"Broadcast plain params over the domain group once at startup (see below)",{"type":48,"tag":581,"props":632,"children":633},{},[634,646,655],{"type":48,"tag":608,"props":635,"children":636},{},[637,639,645],{"type":54,"value":638},"ddp only (",{"type":48,"tag":61,"props":640,"children":642},{"className":641},[],[643],{"type":54,"value":644},"domain=1",{"type":54,"value":620},{"type":48,"tag":608,"props":647,"children":648},{},[649],{"type":48,"tag":61,"props":650,"children":652},{"className":651},[],[653],{"type":54,"value":654},"DistributedDataParallel",{"type":48,"tag":608,"props":656,"children":657},{},[658,660,666],{"type":54,"value":659},"Standard; pass ",{"type":48,"tag":61,"props":661,"children":663},{"className":662},[],[664],{"type":54,"value":665},"process_group=ddp_mesh.get_group()",{"type":54,"value":667}," explicitly, never the default world group",{"type":48,"tag":581,"props":669,"children":670},{},[671,676,684],{"type":48,"tag":608,"props":672,"children":673},{},[674],{"type":54,"value":675},"ddp × domain, params all plain",{"type":48,"tag":608,"props":677,"children":678},{},[679],{"type":48,"tag":61,"props":680,"children":682},{"className":681},[],[683],{"type":54,"value":654},{"type":48,"tag":608,"props":685,"children":686},{},[687],{"type":54,"value":688},"Auto-promotion keeps every param a plain tensor, so ordinary DDP works even combined with domain parallelism",{"type":48,"tag":581,"props":690,"children":691},{},[692,697,708],{"type":48,"tag":608,"props":693,"children":694},{},[695],{"type":54,"value":696},"params sharded (memory) or spatial params as DTensor",{"type":48,"tag":608,"props":698,"children":699},{},[700,702],{"type":54,"value":701},"FSDP2: ",{"type":48,"tag":61,"props":703,"children":705},{"className":704},[],[706],{"type":54,"value":707},"fully_shard(model, mesh=ddp_mesh)",{"type":48,"tag":608,"props":709,"children":710},{},[711],{"type":54,"value":712},"DDP cannot manage DTensor params; FSDP2 shards over exactly the ddp axis (gradients over the domain axis are already reduced by ShardTensor's promotion machinery)",{"type":48,"tag":57,"props":714,"children":715},{},[716,721,722,728,730,736,738,744,746,751,753,759],{"type":48,"tag":86,"props":717,"children":718},{},[719],{"type":54,"value":720},"Never use FSDP1",{"type":54,"value":68},{"type":48,"tag":61,"props":723,"children":725},{"className":724},[],[726],{"type":54,"value":727},"torch.distributed.fsdp.FullyShardedDataParallel",{"type":54,"value":729},",\n",{"type":48,"tag":61,"props":731,"children":733},{"className":732},[],[734],{"type":54,"value":735},"use_orig_params",{"type":54,"value":737},", ",{"type":48,"tag":61,"props":739,"children":741},{"className":740},[],[742],{"type":54,"value":743},"sync_module_states",{"type":54,"value":745},"). It belongs to the old\nDTensor-inheritance era that required ",{"type":48,"tag":61,"props":747,"children":749},{"className":748},[],[750],{"type":54,"value":281},{"type":54,"value":752}," on every parameter,\nfights the auto-promotion design, and is deprecated for this workflow. FSDP2 =\n",{"type":48,"tag":61,"props":754,"children":756},{"className":755},[],[757],{"type":54,"value":758},"torch.distributed.fsdp.fully_shard",{"type":54,"value":760},", always.",{"type":48,"tag":57,"props":762,"children":763},{},[764],{"type":54,"value":765},"Startup sync and FSDP2 specifics:",{"type":48,"tag":330,"props":767,"children":769},{"className":332,"code":768,"language":334,"meta":335,"style":335},"# Neither DDP nor FSDP2 syncs weights over the DOMAIN axis - do it manually\n# whenever domain_size > 1 (before fully_shard for safety):\ngroup = domain_mesh.get_group()\nsrc = torch.distributed.get_global_rank(group, 0)\nwith torch.no_grad():\n    for p in model.parameters():\n        if not isinstance(p, DTensor):\n            torch.distributed.broadcast(p.data, src=src, group=group)\n\n# On the FSDP2 path ONLY: shard statically-shaped spatial params as plain\n# DTensor on the domain mesh (params are static -> DTensor's even chunking is\n# exactly right; ShardTensor is for the possibly-uneven ACTIVATIONS):\nfrom torch.distributed.tensor import distribute_tensor\nmodel.pos_embed = nn.Parameter(\n    distribute_tensor(model.pos_embed.data, domain_mesh, [Shard(1)]))\n# FSDP2 rejects non-contiguous params - make contiguous before fully_shard.\n",[770],{"type":48,"tag":61,"props":771,"children":772},{"__ignoreMap":335},[773,781,789,797,805,813,821,829,837,844,852,860,868,876,884,892],{"type":48,"tag":341,"props":774,"children":775},{"class":343,"line":344},[776],{"type":48,"tag":341,"props":777,"children":778},{},[779],{"type":54,"value":780},"# Neither DDP nor FSDP2 syncs weights over the DOMAIN axis - do it manually\n",{"type":48,"tag":341,"props":782,"children":783},{"class":343,"line":353},[784],{"type":48,"tag":341,"props":785,"children":786},{},[787],{"type":54,"value":788},"# whenever domain_size > 1 (before fully_shard for safety):\n",{"type":48,"tag":341,"props":790,"children":791},{"class":343,"line":362},[792],{"type":48,"tag":341,"props":793,"children":794},{},[795],{"type":54,"value":796},"group = domain_mesh.get_group()\n",{"type":48,"tag":341,"props":798,"children":799},{"class":343,"line":371},[800],{"type":48,"tag":341,"props":801,"children":802},{},[803],{"type":54,"value":804},"src = torch.distributed.get_global_rank(group, 0)\n",{"type":48,"tag":341,"props":806,"children":807},{"class":343,"line":381},[808],{"type":48,"tag":341,"props":809,"children":810},{},[811],{"type":54,"value":812},"with torch.no_grad():\n",{"type":48,"tag":341,"props":814,"children":815},{"class":343,"line":390},[816],{"type":48,"tag":341,"props":817,"children":818},{},[819],{"type":54,"value":820},"    for p in model.parameters():\n",{"type":48,"tag":341,"props":822,"children":823},{"class":343,"line":399},[824],{"type":48,"tag":341,"props":825,"children":826},{},[827],{"type":54,"value":828},"        if not isinstance(p, DTensor):\n",{"type":48,"tag":341,"props":830,"children":831},{"class":343,"line":408},[832],{"type":48,"tag":341,"props":833,"children":834},{},[835],{"type":54,"value":836},"            torch.distributed.broadcast(p.data, src=src, group=group)\n",{"type":48,"tag":341,"props":838,"children":839},{"class":343,"line":416},[840],{"type":48,"tag":341,"props":841,"children":842},{"emptyLinePlaceholder":375},[843],{"type":54,"value":378},{"type":48,"tag":341,"props":845,"children":846},{"class":343,"line":425},[847],{"type":48,"tag":341,"props":848,"children":849},{},[850],{"type":54,"value":851},"# On the FSDP2 path ONLY: shard statically-shaped spatial params as plain\n",{"type":48,"tag":341,"props":853,"children":854},{"class":343,"line":434},[855],{"type":48,"tag":341,"props":856,"children":857},{},[858],{"type":54,"value":859},"# DTensor on the domain mesh (params are static -> DTensor's even chunking is\n",{"type":48,"tag":341,"props":861,"children":862},{"class":343,"line":443},[863],{"type":48,"tag":341,"props":864,"children":865},{},[866],{"type":54,"value":867},"# exactly right; ShardTensor is for the possibly-uneven ACTIVATIONS):\n",{"type":48,"tag":341,"props":869,"children":870},{"class":343,"line":452},[871],{"type":48,"tag":341,"props":872,"children":873},{},[874],{"type":54,"value":875},"from torch.distributed.tensor import distribute_tensor\n",{"type":48,"tag":341,"props":877,"children":878},{"class":343,"line":460},[879],{"type":48,"tag":341,"props":880,"children":881},{},[882],{"type":54,"value":883},"model.pos_embed = nn.Parameter(\n",{"type":48,"tag":341,"props":885,"children":886},{"class":343,"line":469},[887],{"type":48,"tag":341,"props":888,"children":889},{},[890],{"type":54,"value":891},"    distribute_tensor(model.pos_embed.data, domain_mesh, [Shard(1)]))\n",{"type":48,"tag":341,"props":893,"children":894},{"class":343,"line":478},[895],{"type":48,"tag":341,"props":896,"children":897},{},[898],{"type":54,"value":899},"# FSDP2 rejects non-contiguous params - make contiguous before fully_shard.\n",{"type":48,"tag":57,"props":901,"children":902},{},[903,905,911],{"type":54,"value":904},"On the DDP path, leave spatial params plain — auto-promotion handles a\nreplicated pos_embed against sharded activations; do NOT DTensor-shard params\nyou don't have to (a ",{"type":48,"tag":61,"props":906,"children":908},{"className":907},[],[909],{"type":54,"value":910},"Shard",{"type":54,"value":912},"-placement param under DDP breaks DDP).",{"type":48,"tag":57,"props":914,"children":915},{},[916],{"type":54,"value":917},"Reference implementations, in order of usefulness:",{"type":48,"tag":162,"props":919,"children":920},{},[921,953,970],{"type":48,"tag":166,"props":922,"children":923},{},[924,930,932,938,939,945,947],{"type":48,"tag":61,"props":925,"children":927},{"className":926},[],[928],{"type":54,"value":929},"test\u002Fdomain_parallel\u002Fmodels\u002Fharness.py",{"type":54,"value":931}," — ",{"type":48,"tag":61,"props":933,"children":935},{"className":934},[],[936],{"type":54,"value":937},"wrap_ddp",{"type":54,"value":737},{"type":48,"tag":61,"props":940,"children":942},{"className":941},[],[943],{"type":54,"value":944},"shard_spatial_params_",{"type":54,"value":946},"\n(name-based selector for pos_embed\u002FRoPE), ",{"type":48,"tag":61,"props":948,"children":950},{"className":949},[],[951],{"type":54,"value":952},"wrap_fsdp_spatial",{"type":48,"tag":166,"props":954,"children":955},{},[956,962,964],{"type":48,"tag":61,"props":957,"children":959},{"className":958},[],[960],{"type":54,"value":961},"examples\u002Fweather\u002Fstormcast\u002Futils\u002Fparallel.py",{"type":54,"value":963}," — production ",{"type":48,"tag":61,"props":965,"children":967},{"className":966},[],[968],{"type":54,"value":969},"ParallelHelper",{"type":48,"tag":166,"props":971,"children":972},{},[973,979],{"type":48,"tag":61,"props":974,"children":976},{"className":975},[],[977],{"type":54,"value":978},"examples\u002Fminimal\u002FShardTensorExamples\u002F5_vit_training_loop\u002F",{"type":54,"value":980}," — end-to-end\nbenchmark script with DDP\u002FFSDP2\u002Fcompile flags",{"type":48,"tag":57,"props":982,"children":983},{},[984,986,992,994,1000],{"type":54,"value":985},"Optimizer note: ",{"type":48,"tag":61,"props":987,"children":989},{"className":988},[],[990],{"type":54,"value":991},"foreach",{"type":54,"value":993},"-based optimizers (AdamW default) cannot batch plain\ntensors together with DTensors (or DTensors on different meshes) in one param\ngroup. Split param groups by ",{"type":48,"tag":61,"props":995,"children":997},{"className":996},[],[998],{"type":54,"value":999},"p.device_mesh if isinstance(p, DTensor) else None",{"type":54,"value":191},{"type":48,"tag":155,"props":1002,"children":1004},{"id":1003},"torchcompile-with-shardtensor",[1005],{"type":54,"value":1006},"torch.compile with ShardTensor",{"type":48,"tag":162,"props":1008,"children":1009},{},[1010,1051,1076,1087],{"type":48,"tag":166,"props":1011,"children":1012},{},[1013,1018,1020,1026,1028,1034,1036,1041,1043,1049],{"type":48,"tag":86,"props":1014,"children":1015},{},[1016],{"type":54,"value":1017},"Sharded (ring) attention cannot live inside a compiled region",{"type":54,"value":1019}," — see\n",{"type":48,"tag":61,"props":1021,"children":1023},{"className":1022},[],[1024],{"type":54,"value":1025},"physicsnemo\u002Fdomain_parallel\u002Fshard_utils\u002Fattention_patches.py",{"type":54,"value":1027},". With\n",{"type":48,"tag":61,"props":1029,"children":1031},{"className":1030},[],[1032],{"type":54,"value":1033},"domain_size > 1",{"type":54,"value":1035},", compile ",{"type":48,"tag":102,"props":1037,"children":1038},{},[1039],{"type":54,"value":1040},"regionally",{"type":54,"value":1042},": patch-embed \u002F per-block norms and\nMLPs \u002F head, leaving attention eager. With ",{"type":48,"tag":61,"props":1044,"children":1046},{"className":1045},[],[1047],{"type":54,"value":1048},"domain_size == 1",{"type":54,"value":1050},", compile the\nwhole model.",{"type":48,"tag":166,"props":1052,"children":1053},{},[1054,1066,1068,1074],{"type":48,"tag":86,"props":1055,"children":1056},{},[1057,1059,1065],{"type":54,"value":1058},"Pass ",{"type":48,"tag":61,"props":1060,"children":1062},{"className":1061},[],[1063],{"type":54,"value":1064},"dynamic=False",{"type":54,"value":191},{"type":54,"value":1067}," All compiled submodules share dynamo wrapper\nframes; when different submodules (norm vs linear) hit the same frame, the\nrecompile triggers automatic-dynamic, which retraces symbolically and can\nleak SymInts into runtime ",{"type":48,"tag":61,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":54,"value":1073},"ShardTensorSpec",{"type":54,"value":1075},"s. Fixed-shape workloads gain\nnothing from dynamic tracing anyway.",{"type":48,"tag":166,"props":1077,"children":1078},{},[1079,1085],{"type":48,"tag":61,"props":1080,"children":1082},{"className":1081},[],[1083],{"type":54,"value":1084},"torch._dynamo.reset()",{"type":54,"value":1086}," between input-size changes in sweeps.",{"type":48,"tag":166,"props":1088,"children":1089},{},[1090,1092,1097,1099,1105,1107,1113,1115,1121,1123,1129,1131,1137,1139,1144,1146,1152,1154,1160],{"type":54,"value":1091},"Gradients a compiled region returns for a ShardTensor ",{"type":48,"tag":102,"props":1093,"children":1094},{},[1095],{"type":54,"value":1096},"input",{"type":54,"value":1098}," arrive as\nproper ShardTensors. This relies on ",{"type":48,"tag":61,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":54,"value":1104},"torch.autograd.grad",{"type":54,"value":1106}," being in\n",{"type":48,"tag":61,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":54,"value":1112},"_autograd_passthrough_functions",{"type":54,"value":1114},": AOTAutograd's joint trace calls it on\nthe wrapped subclass primals, and routing it through the DTensor fallback\nsevers the graph query (fresh converted tensors + ",{"type":48,"tag":61,"props":1116,"children":1118},{"className":1117},[],[1119],{"type":54,"value":1120},"allow_unused=True",{"type":54,"value":1122}," →\nall-None grads → plain ",{"type":48,"tag":61,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":54,"value":1128},"grad_input_metas",{"type":54,"value":1130},"). If you ever see\n",{"type":48,"tag":61,"props":1132,"children":1134},{"className":1133},[],[1135],{"type":54,"value":1136},"'Tensor' object has no attribute '_local_tensor'",{"type":54,"value":1138}," in an eager backward fed\nby a compiled region, check that passthrough first\n(",{"type":48,"tag":61,"props":1140,"children":1142},{"className":1141},[],[1143],{"type":54,"value":1112},{"type":54,"value":1145}," in\n",{"type":48,"tag":61,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":54,"value":1151},"physicsnemo\u002Fdomain_parallel\u002Fshard_tensor.py",{"type":54,"value":1153},"; regression coverage lives in\n",{"type":48,"tag":61,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":54,"value":1159},"test\u002Fdomain_parallel\u002Ftest_compile.py",{"type":54,"value":1161},", added with the torch.compile\nenablement work — absent on builds that predate it).",{"type":48,"tag":155,"props":1163,"children":1165},{"id":1164},"debugging-pitfalls-each-of-these-cost-real-time-check-them-first",[1166],{"type":54,"value":1167},"Debugging pitfalls (each of these cost real time — check them first)",{"type":48,"tag":1169,"props":1170,"children":1171},"ol",{},[1172,1219,1245,1314,1365,1393],{"type":48,"tag":166,"props":1173,"children":1174},{},[1175,1186,1188,1194,1196,1202,1204,1210,1211,1217],{"type":48,"tag":86,"props":1176,"children":1177},{},[1178,1184],{"type":48,"tag":61,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":54,"value":1183},"TypeError: unsupported operand type(s) for +: 'ShardTensor' and 'ShardTensor'",{"type":54,"value":1185}," is almost never the real error.",{"type":54,"value":1187}," Binary dunders convert an\ninternal ",{"type":48,"tag":61,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":54,"value":1193},"NotImplementedError",{"type":54,"value":1195}," into ",{"type":48,"tag":61,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":54,"value":1201},"NotImplemented",{"type":54,"value":1203},", and CPython emits this\ngeneric message, swallowing the real traceback. Temporarily replace ",{"type":48,"tag":61,"props":1205,"children":1207},{"className":1206},[],[1208],{"type":54,"value":1209},"x + y",{"type":54,"value":129},{"type":48,"tag":61,"props":1212,"children":1214},{"className":1213},[],[1215],{"type":54,"value":1216},"torch.add(x, y)",{"type":54,"value":1218}," to surface the true exception.",{"type":48,"tag":166,"props":1220,"children":1221},{},[1222,1235,1237,1243],{"type":48,"tag":86,"props":1223,"children":1224},{},[1225,1227,1233],{"type":54,"value":1226},"In-place ",{"type":48,"tag":61,"props":1228,"children":1230},{"className":1229},[],[1231],{"type":54,"value":1232},"x.requires_grad_(True)",{"type":54,"value":1234}," on a ShardTensor silently does\nnothing",{"type":54,"value":1236}," — the call routes through the DTensor fallback and sets the flag\non a discarded temporary. Use ",{"type":48,"tag":61,"props":1238,"children":1240},{"className":1239},[],[1241],{"type":54,"value":1242},"scatter_tensor(..., requires_grad=True)",{"type":54,"value":1244}," or\nthread gradients through parameters.",{"type":48,"tag":166,"props":1246,"children":1247},{},[1248,1258,1260,1266,1268,1274,1276,1282,1284,1289,1291,1296,1298,1304,1306,1312],{"type":48,"tag":86,"props":1249,"children":1250},{},[1251,1256],{"type":48,"tag":61,"props":1252,"children":1254},{"className":1253},[],[1255],{"type":54,"value":1104},{"type":54,"value":1257}," works directly on ShardTensors",{"type":54,"value":1259}," — it is an\nautograd-passthrough function (runs on the real tensor objects under\n",{"type":48,"tag":61,"props":1261,"children":1263},{"className":1262},[],[1264],{"type":54,"value":1265},"DisableTorchFunctionSubclass",{"type":54,"value":1267},"). If you see \"not used in the graph\" on a\nShardTensor input, you are on an old build without the passthrough; probe\nwith ",{"type":48,"tag":61,"props":1269,"children":1271},{"className":1270},[],[1272],{"type":54,"value":1273},".backward()",{"type":54,"value":1275}," + ",{"type":48,"tag":61,"props":1277,"children":1279},{"className":1278},[],[1280],{"type":54,"value":1281},"tensor.register_hook(...)",{"type":54,"value":1283}," there instead. Beware\nthat ",{"type":48,"tag":102,"props":1285,"children":1286},{},[1287],{"type":54,"value":1288},"monkeypatching",{"type":54,"value":1290}," ",{"type":48,"tag":61,"props":1292,"children":1294},{"className":1293},[],[1295],{"type":54,"value":1104},{"type":54,"value":1297}," (e.g. to log calls) breaks the\npassthrough: ",{"type":48,"tag":61,"props":1299,"children":1301},{"className":1300},[],[1302],{"type":54,"value":1303},"handle_torch_function",{"type":54,"value":1305}," passes the module-global ",{"type":48,"tag":61,"props":1307,"children":1309},{"className":1308},[],[1310],{"type":54,"value":1311},"grad",{"type":54,"value":1313},"\nresolved at call time, so identity lookups see your wrapper.",{"type":48,"tag":166,"props":1315,"children":1316},{},[1317,1322,1323,1329,1330,1336,1337,1343,1344,1349,1351,1356,1357,1363],{"type":48,"tag":86,"props":1318,"children":1319},{},[1320],{"type":54,"value":1321},"Only certain functions are passthrough-safe",{"type":54,"value":68},{"type":48,"tag":61,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":54,"value":1328},"register_hook",{"type":54,"value":729},{"type":48,"tag":61,"props":1331,"children":1333},{"className":1332},[],[1334],{"type":54,"value":1335},"register_post_accumulate_grad_hook",{"type":54,"value":737},{"type":48,"tag":61,"props":1338,"children":1340},{"className":1339},[],[1341],{"type":54,"value":1342},"retain_grad",{"type":54,"value":729},{"type":48,"tag":61,"props":1345,"children":1347},{"className":1346},[],[1348],{"type":54,"value":1104},{"type":54,"value":1350}," — see ",{"type":48,"tag":61,"props":1352,"children":1354},{"className":1353},[],[1355],{"type":54,"value":1112},{"type":54,"value":1145},{"type":48,"tag":61,"props":1358,"children":1360},{"className":1359},[],[1361],{"type":54,"value":1362},"shard_tensor.py",{"type":54,"value":1364},"). Any other identity-sensitive method may act on a\nconverted temporary.",{"type":48,"tag":166,"props":1366,"children":1367},{},[1368,1370,1375,1377,1383,1385,1391],{"type":54,"value":1369},"Measuring memory\u002Fperf while discarding outputs leaves ",{"type":48,"tag":86,"props":1371,"children":1372},{},[1373],{"type":54,"value":1374},"unwaited async\ncollectives",{"type":54,"value":1376}," (exit-time warnings). Resolve with\n",{"type":48,"tag":61,"props":1378,"children":1380},{"className":1379},[],[1381],{"type":54,"value":1382},"to_local()",{"type":54,"value":1384},"\u002F",{"type":48,"tag":61,"props":1386,"children":1388},{"className":1387},[],[1389],{"type":54,"value":1390},"AsyncCollectiveTensor.wait()",{"type":54,"value":1392}," on discarded results.",{"type":48,"tag":166,"props":1394,"children":1395},{},[1396,1402,1403,1409,1411,1416],{"type":48,"tag":61,"props":1397,"children":1399},{"className":1398},[],[1400],{"type":54,"value":1401},"CommDebugMode",{"type":54,"value":68},{"type":48,"tag":61,"props":1404,"children":1406},{"className":1405},[],[1407],{"type":54,"value":1408},"torch.distributed.tensor.debug",{"type":54,"value":1410},") counts collectives at\ndispatch level — the fastest way to check whether an op path is paying\nhidden communication. A well-supported forward op on sharded activations\nshould show ",{"type":48,"tag":86,"props":1412,"children":1413},{},[1414],{"type":54,"value":1415},"zero",{"type":54,"value":1417}," forward collectives; backward shows domain all-reduces\nfor promoted weight grads (expected and correct).",{"type":48,"tag":155,"props":1419,"children":1421},{"id":1420},"enabling-new-layers-ops",[1422],{"type":54,"value":1423},"Enabling new layers \u002F ops",{"type":48,"tag":57,"props":1425,"children":1426},{},[1427,1429,1435],{"type":54,"value":1428},"Read ",{"type":48,"tag":61,"props":1430,"children":1432},{"className":1431},[],[1433],{"type":54,"value":1434},"references\u002Fnew-op-patterns.md",{"type":54,"value":1436}," before writing any patch. Summary of the\ndecision process:",{"type":48,"tag":1169,"props":1438,"children":1439},{},[1440,1471,1531],{"type":48,"tag":166,"props":1441,"children":1442},{},[1443,1448,1450,1456,1457,1463,1465,1470],{"type":48,"tag":86,"props":1444,"children":1445},{},[1446],{"type":54,"value":1447},"Try the model unmodified first.",{"type":54,"value":1449}," The generic fallback (convert to\nDTensor, run, convert back) covers most ops correctly. Only write a patch\nwhen you observe: a ",{"type":48,"tag":61,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":54,"value":1455},"MissingShardPatch",{"type":54,"value":1384},{"type":48,"tag":61,"props":1458,"children":1460},{"className":1459},[],[1461],{"type":54,"value":1462},"UndeterminedShardingError",{"type":54,"value":1464},", wrong\nnumerics vs a single-GPU run, or unacceptable communication (redistribution\nto Replicate) in ",{"type":48,"tag":61,"props":1466,"children":1468},{"className":1467},[],[1469],{"type":54,"value":1401},{"type":54,"value":191},{"type":48,"tag":166,"props":1472,"children":1473},{},[1474,1476,1481,1483,1489,1491,1497,1499,1505,1507,1513,1515,1521,1523,1529],{"type":54,"value":1475},"Patches are ",{"type":48,"tag":86,"props":1477,"children":1478},{},[1479],{"type":54,"value":1480},"registered from user code at import time",{"type":54,"value":1482}," — no physicsnemo\nfork needed:\n",{"type":48,"tag":61,"props":1484,"children":1486},{"className":1485},[],[1487],{"type":54,"value":1488},"ShardTensor.register_function_handler(torch.nn.functional.foo, wrapper)",{"type":54,"value":1490},"\n(Python\u002F",{"type":48,"tag":61,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":54,"value":1496},"__torch_function__",{"type":54,"value":1498}," level),\n",{"type":48,"tag":61,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":54,"value":1504},"ShardTensor.register_dispatch_handler(aten.foo.default, fn)",{"type":54,"value":1506},"\n(",{"type":48,"tag":61,"props":1508,"children":1510},{"className":1509},[],[1511],{"type":54,"value":1512},"__torch_dispatch__",{"type":54,"value":1514}," level), and\n",{"type":48,"tag":61,"props":1516,"children":1518},{"className":1517},[],[1519],{"type":54,"value":1520},"ShardTensor.register_named_function_handler(\"lib.op.default\", wrapper)",{"type":54,"value":1522},"\nfor ",{"type":48,"tag":61,"props":1524,"children":1526},{"className":1525},[],[1527],{"type":54,"value":1528},"torch.library.custom_op",{"type":54,"value":1530},"s.",{"type":48,"tag":166,"props":1532,"children":1533},{},[1534,1536,1542,1544,1550,1552,1557,1559,1565,1566,1572,1574,1580,1582,1588,1590,1596],{"type":54,"value":1535},"Use the existing patches in ",{"type":48,"tag":61,"props":1537,"children":1539},{"className":1538},[],[1540],{"type":54,"value":1541},"physicsnemo\u002Fdomain_parallel\u002Fshard_utils\u002F",{"type":54,"value":1543}," as\ntemplates: ",{"type":48,"tag":61,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":54,"value":1549},"pooling_patches.py",{"type":54,"value":1551}," (config gating + ",{"type":48,"tag":61,"props":1553,"children":1555},{"className":1554},[],[1556],{"type":54,"value":1455},{"type":54,"value":1558},"),\n",{"type":48,"tag":61,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":54,"value":1564},"conv_patches.py",{"type":54,"value":1275},{"type":48,"tag":61,"props":1567,"children":1569},{"className":1568},[],[1570],{"type":54,"value":1571},"halo.py",{"type":54,"value":1573}," (ops with spatial support needing halo\nexchange), ",{"type":48,"tag":61,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":54,"value":1579},"normalization_patches.py",{"type":54,"value":1581}," (explicit ",{"type":48,"tag":61,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":54,"value":1587},"autograd.Function",{"type":54,"value":1589}," with\ncustom backward), ",{"type":48,"tag":61,"props":1591,"children":1593},{"className":1592},[],[1594],{"type":54,"value":1595},"view_ops.py",{"type":54,"value":1597}," (dual-level registration; shape-only ops).",{"type":48,"tag":155,"props":1599,"children":1601},{"id":1600},"testing-new-layers",[1602],{"type":54,"value":1603},"Testing new layers",{"type":48,"tag":57,"props":1605,"children":1606},{},[1607,1608,1614,1616,1621,1622,1628,1630,1636],{"type":54,"value":1428},{"type":48,"tag":61,"props":1609,"children":1611},{"className":1610},[],[1612],{"type":54,"value":1613},"references\u002Ftesting.md",{"type":54,"value":1615},". The one-line summary: scatter a full input,\nrun the module distributed and single-GPU, and compare outputs ",{"type":48,"tag":102,"props":1617,"children":1618},{},[1619],{"type":54,"value":1620},"and gradients",{"type":54,"value":129},{"type":48,"tag":61,"props":1623,"children":1625},{"className":1624},[],[1626],{"type":54,"value":1627},"numerical_shard_tensor_check(mesh, module, [sharded_x], {}, check_grads=True)",{"type":54,"value":1629}," under the ",{"type":48,"tag":61,"props":1631,"children":1633},{"className":1632},[],[1634],{"type":54,"value":1635},"multigpu_static",{"type":54,"value":1637}," marker, launched as",{"type":48,"tag":330,"props":1639,"children":1643},{"className":1640,"code":1641,"language":1642,"meta":335,"style":335},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","torchrun --nproc-per-node 4 -m pytest test\u002F... --multigpu-static -m multigpu_static\n","bash",[1644],{"type":48,"tag":61,"props":1645,"children":1646},{"__ignoreMap":335},[1647],{"type":48,"tag":341,"props":1648,"children":1649},{"class":343,"line":344},[1650,1656,1662,1668,1673,1678,1683,1688,1692],{"type":48,"tag":341,"props":1651,"children":1653},{"style":1652},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1654],{"type":54,"value":1655},"torchrun",{"type":48,"tag":341,"props":1657,"children":1659},{"style":1658},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1660],{"type":54,"value":1661}," --nproc-per-node",{"type":48,"tag":341,"props":1663,"children":1665},{"style":1664},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1666],{"type":54,"value":1667}," 4",{"type":48,"tag":341,"props":1669,"children":1670},{"style":1658},[1671],{"type":54,"value":1672}," -m",{"type":48,"tag":341,"props":1674,"children":1675},{"style":1658},[1676],{"type":54,"value":1677}," pytest",{"type":48,"tag":341,"props":1679,"children":1680},{"style":1658},[1681],{"type":54,"value":1682}," test\u002F...",{"type":48,"tag":341,"props":1684,"children":1685},{"style":1658},[1686],{"type":54,"value":1687}," --multigpu-static",{"type":48,"tag":341,"props":1689,"children":1690},{"style":1658},[1691],{"type":54,"value":1672},{"type":48,"tag":341,"props":1693,"children":1694},{"style":1658},[1695],{"type":54,"value":1696}," multigpu_static\n",{"type":48,"tag":57,"props":1698,"children":1699},{},[1700,1702,1707,1709,1715],{"type":54,"value":1701},"A forward-only test proves almost nothing — ",{"type":48,"tag":86,"props":1703,"children":1704},{},[1705],{"type":54,"value":1706},"the weight gradient is where\nsharding bugs live",{"type":54,"value":1708}," (it is Partial over the domain mesh and must be reduced).\nAlways ",{"type":48,"tag":61,"props":1710,"children":1712},{"className":1711},[],[1713],{"type":54,"value":1714},"check_grads=True",{"type":54,"value":1716},", always disable TF32 for the comparison.",{"type":48,"tag":155,"props":1718,"children":1720},{"id":1719},"related-resources",[1721],{"type":54,"value":1722},"Related resources",{"type":48,"tag":162,"props":1724,"children":1725},{},[1726,1737,1747,1765],{"type":48,"tag":166,"props":1727,"children":1728},{},[1729,1735],{"type":48,"tag":61,"props":1730,"children":1732},{"className":1731},[],[1733],{"type":54,"value":1734},"references\u002Fintegration-checklist.md",{"type":54,"value":1736}," — step-by-step checklist for\nretrofitting an existing training\u002Finference script, plus the 4-GPU smoke\nmatrix worth scripting.",{"type":48,"tag":166,"props":1738,"children":1739},{},[1740,1745],{"type":48,"tag":61,"props":1741,"children":1743},{"className":1742},[],[1744],{"type":54,"value":1434},{"type":54,"value":1746}," — patch anatomy, registration levels, and\nwhich existing patch to copy for each op class.",{"type":48,"tag":166,"props":1748,"children":1749},{},[1750,1755,1757,1763],{"type":48,"tag":61,"props":1751,"children":1753},{"className":1752},[],[1754],{"type":54,"value":1613},{"type":54,"value":1756}," — multi-GPU test bootstrapping,\n",{"type":48,"tag":61,"props":1758,"children":1760},{"className":1759},[],[1761],{"type":54,"value":1762},"numerical_shard_tensor_check",{"type":54,"value":1764},", markers, and torchrun invocation.",{"type":48,"tag":166,"props":1766,"children":1767},{},[1768,1773],{"type":48,"tag":61,"props":1769,"children":1771},{"className":1770},[],[1772],{"type":54,"value":189},{"type":54,"value":1774}," — for choosing models, datapipes, and examples.",{"type":48,"tag":1776,"props":1777,"children":1778},"style",{},[1779],{"type":54,"value":1780},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1782,"total":1886},[1783,1800,1814,1828,1840,1857,1872],{"slug":1784,"name":1784,"fn":1785,"description":1786,"org":1787,"tags":1788,"stars":23,"repoUrl":24,"updatedAt":1799},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1789,1792,1795,1796],{"name":1790,"slug":1791,"type":15},"Data Analysis","data-analysis",{"name":1793,"slug":1794,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":1797,"slug":1798,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":1801,"name":1801,"fn":1802,"description":1803,"org":1804,"tags":1805,"stars":23,"repoUrl":24,"updatedAt":1813},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1806,1809,1812],{"name":1807,"slug":1808,"type":15},"Deployment","deployment",{"name":1810,"slug":1811,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1815,"name":1815,"fn":1816,"description":1817,"org":1818,"tags":1819,"stars":23,"repoUrl":24,"updatedAt":1827},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1820,1823,1824],{"name":1821,"slug":1822,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1825,"slug":1826,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1829,"name":1829,"fn":1830,"description":1831,"org":1832,"tags":1833,"stars":23,"repoUrl":24,"updatedAt":1839},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1834,1835,1836],{"name":1790,"slug":1791,"type":15},{"name":9,"slug":8,"type":15},{"name":1837,"slug":1838,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":1841,"name":1841,"fn":1842,"description":1843,"org":1844,"tags":1845,"stars":23,"repoUrl":24,"updatedAt":1856},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1846,1849,1852,1853],{"name":1847,"slug":1848,"type":15},"Automation","automation",{"name":1850,"slug":1851,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1854,"slug":1855,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1858,"name":1858,"fn":1859,"description":1860,"org":1861,"tags":1862,"stars":23,"repoUrl":24,"updatedAt":1871},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1863,1864,1867,1868],{"name":1807,"slug":1808,"type":15},{"name":1865,"slug":1866,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1869,"slug":1870,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1873,"name":1873,"fn":1874,"description":1875,"org":1876,"tags":1877,"stars":23,"repoUrl":24,"updatedAt":1885},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1878,1879,1882],{"name":9,"slug":8,"type":15},{"name":1880,"slug":1881,"type":15},"Quantum Computing","quantum-computing",{"name":1883,"slug":1884,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",308,{"items":1888,"total":2036},[1889,1907,1922,1933,1945,1959,1972,1984,1995,2004,2018,2027],{"slug":1890,"name":1890,"fn":1891,"description":1892,"org":1893,"tags":1894,"stars":1904,"repoUrl":1905,"updatedAt":1906},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1895,1898,1901],{"name":1896,"slug":1897,"type":15},"Documentation","documentation",{"name":1899,"slug":1900,"type":15},"MCP","mcp",{"name":1902,"slug":1903,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":1908,"name":1908,"fn":1909,"description":1910,"org":1911,"tags":1912,"stars":1919,"repoUrl":1920,"updatedAt":1921},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1913,1916,1917],{"name":1914,"slug":1915,"type":15},"Containers","containers",{"name":1807,"slug":1808,"type":15},{"name":1918,"slug":334,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1923,"name":1923,"fn":1924,"description":1925,"org":1926,"tags":1927,"stars":1919,"repoUrl":1920,"updatedAt":1932},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1928,1931],{"name":1929,"slug":1930,"type":15},"CI\u002FCD","ci-cd",{"name":1807,"slug":1808,"type":15},"2026-07-14T05:25:59.97109",{"slug":1934,"name":1934,"fn":1935,"description":1936,"org":1937,"tags":1938,"stars":1919,"repoUrl":1920,"updatedAt":1944},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1939,1940,1941],{"name":1929,"slug":1930,"type":15},{"name":1807,"slug":1808,"type":15},{"name":1942,"slug":1943,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":1946,"name":1946,"fn":1947,"description":1948,"org":1949,"tags":1950,"stars":1919,"repoUrl":1920,"updatedAt":1958},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1951,1954,1955],{"name":1952,"slug":1953,"type":15},"Debugging","debugging",{"name":1942,"slug":1943,"type":15},{"name":1956,"slug":1957,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1960,"name":1960,"fn":1961,"description":1962,"org":1963,"tags":1964,"stars":1919,"repoUrl":1920,"updatedAt":1971},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1965,1968],{"name":1966,"slug":1967,"type":15},"Best Practices","best-practices",{"name":1969,"slug":1970,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1973,"name":1973,"fn":1974,"description":1975,"org":1976,"tags":1977,"stars":1919,"repoUrl":1920,"updatedAt":1983},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1978,1979,1982],{"name":17,"slug":18,"type":15},{"name":1980,"slug":1981,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":1985,"name":1985,"fn":1986,"description":1987,"org":1988,"tags":1989,"stars":1919,"repoUrl":1920,"updatedAt":1994},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1990,1993],{"name":1991,"slug":1992,"type":15},"QA","qa",{"name":1837,"slug":1838,"type":15},"2026-07-14T05:25:53.673039",{"slug":1996,"name":1996,"fn":1997,"description":1998,"org":1999,"tags":2000,"stars":1919,"repoUrl":1920,"updatedAt":2003},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2001,2002],{"name":1807,"slug":1808,"type":15},{"name":1810,"slug":1811,"type":15},"2026-07-14T05:25:49.362534",{"slug":2005,"name":2005,"fn":2006,"description":2007,"org":2008,"tags":2009,"stars":1919,"repoUrl":1920,"updatedAt":2017},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2010,2013,2014],{"name":2011,"slug":2012,"type":15},"Code Review","code-review",{"name":1942,"slug":1943,"type":15},{"name":2015,"slug":2016,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":2019,"name":2019,"fn":2020,"description":2021,"org":2022,"tags":2023,"stars":1919,"repoUrl":1920,"updatedAt":2026},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2024,2025],{"name":1991,"slug":1992,"type":15},{"name":1837,"slug":1838,"type":15},"2026-07-14T05:25:54.928983",{"slug":2028,"name":2028,"fn":2029,"description":2030,"org":2031,"tags":2032,"stars":1919,"repoUrl":1920,"updatedAt":2035},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2033,2034],{"name":1847,"slug":1848,"type":15},{"name":1929,"slug":1930,"type":15},"2026-07-30T05:29:03.275638",563]