[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-warp-debug-gradients":3,"mdc--iy0gpe-key":40,"related-repo-nvidia-warp-debug-gradients":1181,"related-org-nvidia-warp-debug-gradients":1214},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":35,"sourceUrl":38,"mdContent":39},"warp-debug-gradients","debug gradients in Warp programs","Use to diagnose and fix incorrect gradients in differentiable Warp programs. Anything trained, optimized, calibrated, or fit through Warp kernels depends on wp.Tape gradients, so treat any misbehavior of such a workflow as a gradient problem until proven otherwise — use this when training diverges or NaNs, won't train at all, stalls or plateaus above the expected loss, converges to a wrong or biased answer, is worse than a reference implementation, works at small scale but fails at production scale, or fails a QA\u002Fvalidation recheck. Also for explicit symptoms — exploding, NaN\u002Finf, zero, or subtly wrong gradients, suspected wp.Tape\u002Fbackward issues, gradcheck failures — but users usually describe only the surface symptom (\"the sim explodes\", \"the fit gets dragged toward outliers\") without mentioning gradients: make that leap. Not for forward-only Warp work, build\u002Finstall problems, or autograd issues in other frameworks without Warp.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Engineering","engineering",{"name":21,"slug":22,"type":15},"Debugging","debugging",6864,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fwarp","2026-08-05T05:58:27.639091","Apache-2.0",555,[29,30,31,32,8,33,34],"cuda","differentiable-programming","gpu","gpu-acceleration","nvidia-warp","python",{"repoUrl":24,"stars":23,"forks":27,"topics":36,"description":37},[29,30,31,32,8,33,34],"A Python framework for GPU-accelerated simulation, robotics, and machine learning.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fwarp\u002Ftree\u002FHEAD\u002Fskills\u002Fwarp-debug-gradients","---\nname: warp-debug-gradients\ndescription: >-\n  Use to diagnose and fix incorrect gradients in differentiable Warp programs.\n  Anything trained, optimized, calibrated, or fit through Warp kernels depends\n  on wp.Tape gradients, so treat any misbehavior of such a workflow as a\n  gradient problem until proven otherwise — use this when training diverges or\n  NaNs, won't train at all, stalls or plateaus above the expected loss,\n  converges to a wrong or biased answer, is worse than a reference\n  implementation, works at small scale but fails at production scale, or fails\n  a QA\u002Fvalidation recheck. Also for explicit symptoms — exploding, NaN\u002Finf,\n  zero, or subtly wrong gradients, suspected wp.Tape\u002Fbackward issues, gradcheck\n  failures — but users usually describe only the surface symptom (\"the sim\n  explodes\", \"the fit gets dragged toward outliers\") without mentioning\n  gradients: make that leap. Not for forward-only Warp work, build\u002Finstall\n  problems, or autograd issues in other frameworks without Warp.\nlicense: Apache-2.0\ncompatibility: Requires a working NVIDIA Warp installation (>= 1.13 minimum; >= 1.17 recommended for reliable verification — copy-adjoint accumulation, overwrite-warning call sites, read-flag lifetime, and gradcheck's restore_inputs changed in 1.17 and are version-caveated in the references; on older versions a fixed bug class still exists and some tools need workarounds). Diagnosis runs the user's reproduction, so a functioning device (CPU or CUDA) is needed.\nmetadata:\n  author: \"Warp Team \u003Cwarp-python@nvidia.com>\"\n  version: \"0.1.0\"\n  tags:\n  - warp\n  - autodiff\n  - gradients\n  - differentiable-simulation\n  - debugging\n  upstream: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fwarp\n---\n\n# Debugging Gradients in Warp\n\nGradient bugs in Warp are almost never math bugs. The forward simulation looks\nperfectly healthy while the backward pass silently reads clobbered values,\nskips arrays, or double-counts adjoints. Users routinely burn days tuning\nphysics knobs, loss functions, and assets when the real cause is a two-line\ntaping-pattern fix. Your job is to find that fix with evidence, not intuition.\n\nThe single most important discipline: **measure before hypothesizing**. It is\ncheap for you to run a shrunk reproduction and compare autodiff against finite\ndifferences. The *way* the gradient is wrong (its signature) prunes the\nhypothesis space far faster than reading code ever will. Do not start\nproposing fixes from code reading alone — plausible-looking diagnoses of\ndifferentiability bugs are very often wrong, and an unverified \"fix\" that\nhappens to perturb the numbers wastes everyone's time.\n\n## When to Use This Skill\n\nAnything trained, optimized, calibrated, or fit through Warp kernels flows\nthrough `wp.Tape` gradients — so when such a workflow misbehaves, gradients\nare the prime suspect even if the user never says the word. Activate on the\nsymptoms users actually report: training that diverges, NaNs, or does\nnothing; loss that stalls or plateaus above where it should; fits that\nconverge to a wrong or biased answer or are worse than a reference\nimplementation; pipelines that work at small scale but fail at production\nscale or fail a QA recheck. Also activate on explicit gradient symptoms —\nexploding, NaN\u002Finf, zero, or subtly wrong gradients,\n`wp.autograd.gradcheck` failures, suspected `wp.Tape`\u002Fbackward issues — and\nwhen the user asks whether their gradients can be trusted.\n\nDo not activate for forward-only Warp work (kernel authoring, rendering,\nperformance tuning), Warp build or installation problems, autograd questions\nin other frameworks with no Warp involvement, or pure performance work on a\nbackward pass whose gradients the user has already validated.\n\nThe canonical background is Warp's own documentation — consult the relevant\nsection before diagnosing in its territory (online at\nhttps:\u002F\u002Fnvidia.github.io\u002Fwarp\u002Fstable\u002F; in a Warp source checkout the same content\nis under `docs\u002Fuser_guide\u002F`; pip installs do not include it):\n\n- The \"Differentiability\" guide — especially \"Array Overwrites\", \"Debugging\n  Gradients\", \"Array Overwrite Tracking\", and \"Limitations and Workarounds\"\n  (in-place math, component assignment, dynamic loops).\n- The FAQ, section \"Differentiation and Interoperability\" — what state a\n  tape does and does not preserve, and checkpointing.\n\n## Prerequisites\n\nExecuting this skill assumes all of the following; if one is missing,\nsurface that to the user instead of improvising around it:\n\n- The user's script (or a faithful reproduction) is available in the\n  workspace, runnable, and modifiable — diagnosis executes it repeatedly and\n  edits it to apply fixes.\n- The agent can execute Python with a working Warp install on a usable\n  device (CPU suffices for most diagnosis), including the verification\n  tooling: `wp.autograd.gradcheck`, `wp.autograd.gradcheck_tape`, and the\n  overwrite tracker.\n- This skill's `references\u002F` files (quick-checks.md, verification.md,\n  custom-gradients.md, case-studies.md) accompany it and are consulted at\n  the steps that cite them.\n\n## Instructions\n\n1. **Note the user's Warp version first** (`wp.__version__` or the banner\n   Warp prints at init). Several verification behaviors changed in Warp\n   1.17 — copy-adjoint accumulation, overwrite-warning call sites, read-flag\n   lifetime, `gradcheck`'s `restore_inputs` — and the references mark each\n   with a version caveat. On Warp \u003C 1.17, a whole bug class exists that\n   later versions fixed (quick-checks §1's version caveat), and some tools\n   need workarounds.\n\n2. **Reproduce and shrink.** Get the user's script running (if Warp is not\n   installed, bootstrap non-destructively: create a fresh virtual\n   environment — `python3 -m venv` or `uv venv` — rather than deleting an\n   existing one or installing with `--break-system-packages`), then cut it\n   down:\n   fewer particles\u002Felements, fewer time steps, fewer optimizer iterations,\n   CPU device if the sim allows. You need a repro that runs in seconds,\n   because you will run it many times. Keep the structure (number of kernels,\n   the taping pattern, buffer reuse) intact — that is where the bug lives.\n   Shrinking the *physics* is fine; restructuring the *dataflow* is not.\n   If the script cannot be made to run after non-destructive setup (missing\n   dependencies, broken code), report the blocking issue as the deliverable\n   and stop — do not proceed to verify a program that never ran.\n\n3. **Instrument and establish ground truth** (details and templates in\n   `references\u002Fverification.md`):\n   - Set `wp.config.verify_autograd_array_access = True` before module load\n     and rerun under an active tape. Capture every warning. This catches the\n     single most common bug class (write-after-read overwrites) nearly for\n     free. Know its blind spots: it needs a tape, it cannot see arrays stored\n     inside Warp structs, and it disables kernel caching (expect a kernel\n     rebuild — JIT module recompilation only, not a rebuild of the native\n     library). If the tracker runs clean but gradients are still wrong,\n     specifically check for in-place mutations of arrays held inside Warp\n     structs (quick-checks §1 and Limitations) before trusting the clean\n     result.\n   - Run one **end-to-end finite-difference check**: wrap the full forward\n     pass (sim steps + loss) in a Python callable and hand it to\n     `wp.autograd.gradcheck` with the true optimization inputs — it compares\n     the autodiff gradient against central differences, restoring array\n     inputs between evaluations (Warp 1.17+) so in-place-mutating forwards\n     are checked from pristine state; on older Warp use the manual harness\n     in `references\u002Fverification.md`. The reference is the user's *actual\n     objective over\n     the full horizon*, compared against the gradient the optimizer *actually\n     consumes* — never a narrower window (see `references\u002Fverification.md`).\n     This confirms gradients are actually wrong (users are sometimes wrong\n     about this — report \"gradients are correct\" findings honestly) and\n     yields the error signature. The template in\n     `references\u002Fverification.md` fixes the eps\u002Ftolerance choices and the\n     seed-pinning a stochastic forward needs — do not eyeball pass\u002Ffail\n     against floating-point or sampling noise. If the backward pass runs out\n     of memory while establishing ground truth, apply the checkpointing\n     pattern from \"Edge case: out of memory\" below before proceeding.\n\n4. **Match the signature** against the table below to rank hypotheses.\n\n5. **Scan the code against the known-pattern checklist**\n   (`references\u002Fquick-checks.md`). This is fast for you — do it in the same\n   pass, but let the signature decide which findings are plausible causes\n   versus incidental smells.\n\n6. **Localize if still ambiguous.** Binary-search the pipeline: truncate to K\n   steps and find where FD and autodiff first diverge; run\n   `wp.autograd.gradcheck_tape` to test each recorded launch in isolation.\n   Remember gradcheck_tape validates kernels *individually* — it is\n   structurally blind to inter-kernel overwrites, so a clean per-kernel pass\n   plus a wrong end-to-end gradient points *at* the taping pattern, not the\n   kernels. It also silently *skips* kernels compiled with\n   `enable_backward=False` (see Limitations) — if any kernel in the pipeline\n   sets that, a clean pass says nothing about it; verify it separately.\n\n7. **Fix minimally, then re-verify** with the exact same FD harness that\n   established the failure. A gradient fix without a before\u002Fafter FD\n   comparison is not a fix. Verify the exact program you are shipping — the\n   fixed file as it stands, every line included — never a re-implementation\n   of it in a diagnostic script: a rebuilt pipeline silently drops whatever\n   you believed was irrelevant, and if that belief is wrong the verification\n   passes while the shipped code stays broken. Mechanically: the harness\n   must *import the fixed module (or execute the fixed file) and call into\n   it* — the only code that may live outside the shipped program is the FD\n   driver itself. Also rerun the overwrite\n   tracker to confirm the warnings are gone. \"Minimally\" applies to the code\n   diff, not the diagnosis:\n   when the root cause is structural (e.g., accidental gradient truncation,\n   quick-checks §8), the minimal *correct* fix is the restructure — do not\n   substitute a smaller change that only silences the surface symptom.\n\n8. **Close the loop on the user's original complaint.** Rerun their actual\n   workflow (their script, their printed metrics). The job is done when the\n   symptom they reported is resolved — an optimization that was \"exploding\"\n   should now demonstrably *improve its objective*, not merely avoid NaN. If\n   gradients verify correct at the full horizon but training still fails,\n   that is a new signature-table entry, not a victory; keep diagnosing (or\n   report the verified gradients and the remaining non-gradient cause, e.g.\n   learning rate).\n\n## Failure signatures\n\n| Signature | Leading hypotheses |\n|---|---|\n| Gradients exactly zero | Missing `requires_grad=True` somewhere in the chain (note `wp.zeros` defaults to `False`; `zeros_like`\u002F`clone` inherit from source); `enable_backward=False` at module\u002Fkernel level; loss array not connected to the tape; grads read after `tape.zero()`; a piecewise-constant op (`round`\u002F`floor`\u002F`sign`\u002Fcast\u002Fthreshold) in the chain — there zero is *correct* and the fix is a surrogate gradient such as a straight-through estimator, not a bug hunt (quick-checks §9c); on Warp \u003C 1.17, a tape-recorded copy\u002Fclone whose source has other downstream readers (see the version caveat in `references\u002Fquick-checks.md`) |\n| Gradients grow without bound across optimizer iterations | Missing `tape.zero()`\u002F`tape.reset()` between iterations; state-object aliasing that carries an in-tape overwrite across frames (case study 1) |\n| Off by an exact small factor (2x, Nx) | Double accumulation: a duplicate launch recorded on the tape — note that since Warp 1.13 the store adjoint consumes the output gradient on first use, so a bare duplicate is inert unless the rewritten array has `retain_grad=True` (quick-checks §7) or the Warp version is older; overlapping tape scopes taping the same work twice. Also: a backward seed that does not match the stated objective — seeding a per-element loss adjoint with ones backpropagates the *sum*, exactly N× the *mean* objective's gradient |\n| NaN or inf | Non-differentiable point evaluated in the backward pass (`wp.sqrt(0)`, `wp.length(0)`, `wp.normalize(0)`, division) — needs a custom gradient (`references\u002Fcustom-gradients.md`) or, better, a stable reformulation; an overflow evaluated in the *unselected* branch of `wp.where` (a select, not a branch — quick-checks §9b); dynamic-loop local not recomputed during replay (documented to produce `inf`) |\n| Subtly wrong, often worse with more steps\u002Fiterations | Write-after-read overwrite: `wp.copy` onto an already-read array, ping-pong buffers within one tape, Python rebinding that aliases two \"different\" states (case studies); in-place `*=`\u002F`\u002F=`; vector\u002Fmatrix component reassignment; dynamic-loop intermediates; on Warp \u003C 1.17, a recorded copy\u002Fclone that is not the last consumer of its source (version caveat in `references\u002Fquick-checks.md`) |\n| Per-window FD agrees but full-horizon FD disagrees; or gradients \"verified\" yet the optimizer stalls or worsens the loss | **Accidental gradient truncation**: a tape-per-step loop with backward inside it and state carried between tapes optimizes a different objective than the one being reported (see quick-checks §8). The structural fix is one tape over the whole horizon with `total_steps + 1` distinct state buffers. The solver-space analog: a partially converged iterative solve inside the tape makes FD and autodiff agree on the wrong program — converge it outside the tape and warm-start the taped iterations (quick-checks §8) |\n| Gradients disagree (vs a reference implementation or run-to-run) only on a sparse, data-dependent subset; forward outputs match to float precision | Under-determined forward choice at a non-smooth point (quick-checks §9): both answers can be valid subgradients, and FD cannot adjudicate at a kink. Check whether the discrete choice differs at exactly the mismatching elements before hunting corruption |\n| FD and autodiff agree *at the full horizon* but optimization still fails | Not a gradient bug. Say so. Look at learning rate, loss landscape, physics stability — and report the verified-correct gradients as the finding |\n\n## Examples\n\nA representative session, end to end. A user reports \"my cloth sim trains for\na while, then the loss creeps back up — tuning the learning rate doesn't\nhelp.\" No mention of gradients; the leap is made because the workflow\noptimizes through Warp kernels.\n\n1. Their script runs 512 particles for 200 steps per iteration. Shrink to 16\n   particles, 10 steps, CPU — repro now runs in ~2 s and shows the same\n   creep.\n2. `wp.config.verify_autograd_array_access = True` under the tape prints:\n   `array ... was read from kernel integrate and is now being written to by\n   kernel integrate` — a write-after-read overwrite.\n3. End-to-end `wp.autograd.gradcheck` on the shrunk repro: max relative error\n   0.4 against finite differences. Gradients are confirmed wrong, with the\n   \"subtly wrong, worse with more steps\" signature.\n4. The signature row plus quick-checks §1 point at buffer reuse inside one\n   tape: the sim steps `state_a → state_b → state_a`, ping-ponging two\n   buffers, so the backward pass reads clobbered states.\n5. Minimal fix: allocate `num_steps + 1` distinct state buffers recorded on\n   the tape (physics untouched; only the dataflow changes).\n6. Re-verify: same gradcheck harness now passes (max relative error 3e-4);\n   the overwrite warning is gone; the user's full-size training run now\n   decreases monotonically.\n\nReport: root cause (in-tape buffer reuse), the evidence chain (warning +\nbefore\u002Fafter FD numbers), the two-line diff, and a pointer to the\n\"Array Overwrites\" section of the Differentiability guide.\n\n## Reporting\n\nLead with the root cause and the evidence chain: the FD-vs-autodiff numbers\nthat established the failure, the warning or localization step that found the\ncause, the minimal diff, and the FD numbers after the fix. Name the\ndocumentation section that covers the pattern so the user can read the\ncanonical explanation. If you checked patterns that came up clean (e.g., the\noverwrite tracker found nothing), say so — it tells the user what has been\nruled out.\n\nIf the user is only asking *whether* their gradients are trustworthy, stop\nafter verification and report; apply fixes when they ask for fixes.\n\nPreserve the evidence: leave the diagnostic scripts (FD harness, shrunk\nrepro) in the workspace and list them in the report instead of deleting\nthem — they are the reproducible half of the evidence chain, and the user\nor a reviewer should be able to rerun the exact verification that\njustified the fix. Never delete files you did not create.\n\n## Limitations\n\nThe verification tooling has blind spots — a clean pass through any one\ntool is not a clean bill of health (details in\n`references\u002Fverification.md`):\n\n- The overwrite tracker requires an active tape, cannot see arrays stored\n  inside Warp structs, and disables kernel caching while enabled.\n- `wp.autograd.gradcheck` does not accept struct inputs; wrap the forward\n  in a callable over the underlying arrays. On Warp \u003C 1.17 it does not\n  restore mutated array inputs between evaluations (use the manual\n  harness).\n- `wp.autograd.gradcheck_tape` validates each recorded launch in\n  isolation — it is structurally blind to inter-kernel overwrite bugs and\n  silently skips kernels compiled with `enable_backward=False`.\n- The `*=`\u002F`\u002F=` non-differentiability warning is emitted only at codegen\n  time under `wp.LOG_DEBUG`, so its absence from a normal run means\n  nothing.\n- Warp has no built-in gradient checkpointing; long-horizon memory\n  pressure needs the application-level pattern below.\n- At non-smooth points (ties, kinks, argmin selections), finite\n  differences cannot adjudicate between valid subgradients — FD-vs-AD\n  disagreement there is not automatically a bug (quick-checks §9).\n\n## Edge case: out of memory\n\nIf the backward pass fails to allocate (long simulations keep every\nintermediate state alive on the tape), the fix is gradient checkpointing:\nsave periodic states, replay the segments between them during backward. Warp\nhas no built-in utility — applications implement it themselves. Use\n`warp\u002Fexamples\u002Foptim\u002Fexample_fluid_checkpoint.py` as the reference pattern,\nand see the FAQ's \"Differentiation and Interoperability\" section.\n\n## Reference files\n\n- `references\u002Fquick-checks.md` — the known-bug-pattern checklist with doc\n  pointers and the caveats that make each pattern easy to miss.\n- `references\u002Fverification.md` — tooling details: overwrite tracker setup and\n  blind spots, end-to-end FD harness template, `wp.autograd`\n  gradcheck\u002Fjacobian usage and caveats, tape visualization, bisection.\n- `references\u002Fcustom-gradients.md` — `@wp.func_grad`, `@wp.func_replay`,\n  `@wp.func_native`: when they are required and how they are misused.\n- `references\u002Fcase-studies.md` — two real debugging sagas (state aliasing;\n  differentiable-copy overwrite) showing how subtle the surface symptoms are.\n  Read these when the checklist comes up clean — they calibrate what \"subtle\"\n  means here.\n",{"data":41,"body":51},{"name":4,"description":6,"license":26,"compatibility":42,"metadata":43},"Requires a working NVIDIA Warp installation (>= 1.13 minimum; >= 1.17 recommended for reliable verification — copy-adjoint accumulation, overwrite-warning call sites, read-flag lifetime, and gradcheck's restore_inputs changed in 1.17 and are version-caveated in the references; on older versions a fixed bug class still exists and some tools need workarounds). Diagnosis runs the user's reproduction, so a functioning device (CPU or CUDA) is needed.",{"author":44,"version":45,"tags":46,"upstream":24},"Warp Team \u003Cwarp-python@nvidia.com>","0.1.0",[47,48,49,50,22],"warp","autodiff","gradients","differentiable-simulation",{"type":52,"children":53},"root",[54,63,69,90,97,126,131,154,169,175,180,221,227,516,522,880,886,891,960,965,971,976,988,993,999,1010,1081,1087,1100,1106],{"type":55,"tag":56,"props":57,"children":59},"element","h1",{"id":58},"debugging-gradients-in-warp",[60],{"type":61,"value":62},"text","Debugging Gradients in Warp",{"type":55,"tag":64,"props":65,"children":66},"p",{},[67],{"type":61,"value":68},"Gradient bugs in Warp are almost never math bugs. The forward simulation looks\nperfectly healthy while the backward pass silently reads clobbered values,\nskips arrays, or double-counts adjoints. Users routinely burn days tuning\nphysics knobs, loss functions, and assets when the real cause is a two-line\ntaping-pattern fix. Your job is to find that fix with evidence, not intuition.",{"type":55,"tag":64,"props":70,"children":71},{},[72,74,80,82,88],{"type":61,"value":73},"The single most important discipline: ",{"type":55,"tag":75,"props":76,"children":77},"strong",{},[78],{"type":61,"value":79},"measure before hypothesizing",{"type":61,"value":81},". It is\ncheap for you to run a shrunk reproduction and compare autodiff against finite\ndifferences. The ",{"type":55,"tag":83,"props":84,"children":85},"em",{},[86],{"type":61,"value":87},"way",{"type":61,"value":89}," the gradient is wrong (its signature) prunes the\nhypothesis space far faster than reading code ever will. Do not start\nproposing fixes from code reading alone — plausible-looking diagnoses of\ndifferentiability bugs are very often wrong, and an unverified \"fix\" that\nhappens to perturb the numbers wastes everyone's time.",{"type":55,"tag":91,"props":92,"children":94},"h2",{"id":93},"when-to-use-this-skill",[95],{"type":61,"value":96},"When to Use This Skill",{"type":55,"tag":64,"props":98,"children":99},{},[100,102,109,111,117,119,124],{"type":61,"value":101},"Anything trained, optimized, calibrated, or fit through Warp kernels flows\nthrough ",{"type":55,"tag":103,"props":104,"children":106},"code",{"className":105},[],[107],{"type":61,"value":108},"wp.Tape",{"type":61,"value":110}," gradients — so when such a workflow misbehaves, gradients\nare the prime suspect even if the user never says the word. Activate on the\nsymptoms users actually report: training that diverges, NaNs, or does\nnothing; loss that stalls or plateaus above where it should; fits that\nconverge to a wrong or biased answer or are worse than a reference\nimplementation; pipelines that work at small scale but fail at production\nscale or fail a QA recheck. Also activate on explicit gradient symptoms —\nexploding, NaN\u002Finf, zero, or subtly wrong gradients,\n",{"type":55,"tag":103,"props":112,"children":114},{"className":113},[],[115],{"type":61,"value":116},"wp.autograd.gradcheck",{"type":61,"value":118}," failures, suspected ",{"type":55,"tag":103,"props":120,"children":122},{"className":121},[],[123],{"type":61,"value":108},{"type":61,"value":125},"\u002Fbackward issues — and\nwhen the user asks whether their gradients can be trusted.",{"type":55,"tag":64,"props":127,"children":128},{},[129],{"type":61,"value":130},"Do not activate for forward-only Warp work (kernel authoring, rendering,\nperformance tuning), Warp build or installation problems, autograd questions\nin other frameworks with no Warp involvement, or pure performance work on a\nbackward pass whose gradients the user has already validated.",{"type":55,"tag":64,"props":132,"children":133},{},[134,136,144,146,152],{"type":61,"value":135},"The canonical background is Warp's own documentation — consult the relevant\nsection before diagnosing in its territory (online at\n",{"type":55,"tag":137,"props":138,"children":142},"a",{"href":139,"rel":140},"https:\u002F\u002Fnvidia.github.io\u002Fwarp\u002Fstable\u002F",[141],"nofollow",[143],{"type":61,"value":139},{"type":61,"value":145},"; in a Warp source checkout the same content\nis under ",{"type":55,"tag":103,"props":147,"children":149},{"className":148},[],[150],{"type":61,"value":151},"docs\u002Fuser_guide\u002F",{"type":61,"value":153},"; pip installs do not include it):",{"type":55,"tag":155,"props":156,"children":157},"ul",{},[158,164],{"type":55,"tag":159,"props":160,"children":161},"li",{},[162],{"type":61,"value":163},"The \"Differentiability\" guide — especially \"Array Overwrites\", \"Debugging\nGradients\", \"Array Overwrite Tracking\", and \"Limitations and Workarounds\"\n(in-place math, component assignment, dynamic loops).",{"type":55,"tag":159,"props":165,"children":166},{},[167],{"type":61,"value":168},"The FAQ, section \"Differentiation and Interoperability\" — what state a\ntape does and does not preserve, and checkpointing.",{"type":55,"tag":91,"props":170,"children":172},{"id":171},"prerequisites",[173],{"type":61,"value":174},"Prerequisites",{"type":55,"tag":64,"props":176,"children":177},{},[178],{"type":61,"value":179},"Executing this skill assumes all of the following; if one is missing,\nsurface that to the user instead of improvising around it:",{"type":55,"tag":155,"props":181,"children":182},{},[183,188,208],{"type":55,"tag":159,"props":184,"children":185},{},[186],{"type":61,"value":187},"The user's script (or a faithful reproduction) is available in the\nworkspace, runnable, and modifiable — diagnosis executes it repeatedly and\nedits it to apply fixes.",{"type":55,"tag":159,"props":189,"children":190},{},[191,193,198,200,206],{"type":61,"value":192},"The agent can execute Python with a working Warp install on a usable\ndevice (CPU suffices for most diagnosis), including the verification\ntooling: ",{"type":55,"tag":103,"props":194,"children":196},{"className":195},[],[197],{"type":61,"value":116},{"type":61,"value":199},", ",{"type":55,"tag":103,"props":201,"children":203},{"className":202},[],[204],{"type":61,"value":205},"wp.autograd.gradcheck_tape",{"type":61,"value":207},", and the\noverwrite tracker.",{"type":55,"tag":159,"props":209,"children":210},{},[211,213,219],{"type":61,"value":212},"This skill's ",{"type":55,"tag":103,"props":214,"children":216},{"className":215},[],[217],{"type":61,"value":218},"references\u002F",{"type":61,"value":220}," files (quick-checks.md, verification.md,\ncustom-gradients.md, case-studies.md) accompany it and are consulted at\nthe steps that cite them.",{"type":55,"tag":91,"props":222,"children":224},{"id":223},"instructions",[225],{"type":61,"value":226},"Instructions",{"type":55,"tag":228,"props":229,"children":230},"ol",{},[231,265,313,401,411,429,475,499],{"type":55,"tag":159,"props":232,"children":233},{},[234,239,241,247,249,255,257,263],{"type":55,"tag":75,"props":235,"children":236},{},[237],{"type":61,"value":238},"Note the user's Warp version first",{"type":61,"value":240}," (",{"type":55,"tag":103,"props":242,"children":244},{"className":243},[],[245],{"type":61,"value":246},"wp.__version__",{"type":61,"value":248}," or the banner\nWarp prints at init). Several verification behaviors changed in Warp\n1.17 — copy-adjoint accumulation, overwrite-warning call sites, read-flag\nlifetime, ",{"type":55,"tag":103,"props":250,"children":252},{"className":251},[],[253],{"type":61,"value":254},"gradcheck",{"type":61,"value":256},"'s ",{"type":55,"tag":103,"props":258,"children":260},{"className":259},[],[261],{"type":61,"value":262},"restore_inputs",{"type":61,"value":264}," — and the references mark each\nwith a version caveat. On Warp \u003C 1.17, a whole bug class exists that\nlater versions fixed (quick-checks §1's version caveat), and some tools\nneed workarounds.",{"type":55,"tag":159,"props":266,"children":267},{},[268,273,275,281,283,289,291,297,299,304,306,311],{"type":55,"tag":75,"props":269,"children":270},{},[271],{"type":61,"value":272},"Reproduce and shrink.",{"type":61,"value":274}," Get the user's script running (if Warp is not\ninstalled, bootstrap non-destructively: create a fresh virtual\nenvironment — ",{"type":55,"tag":103,"props":276,"children":278},{"className":277},[],[279],{"type":61,"value":280},"python3 -m venv",{"type":61,"value":282}," or ",{"type":55,"tag":103,"props":284,"children":286},{"className":285},[],[287],{"type":61,"value":288},"uv venv",{"type":61,"value":290}," — rather than deleting an\nexisting one or installing with ",{"type":55,"tag":103,"props":292,"children":294},{"className":293},[],[295],{"type":61,"value":296},"--break-system-packages",{"type":61,"value":298},"), then cut it\ndown:\nfewer particles\u002Felements, fewer time steps, fewer optimizer iterations,\nCPU device if the sim allows. You need a repro that runs in seconds,\nbecause you will run it many times. Keep the structure (number of kernels,\nthe taping pattern, buffer reuse) intact — that is where the bug lives.\nShrinking the ",{"type":55,"tag":83,"props":300,"children":301},{},[302],{"type":61,"value":303},"physics",{"type":61,"value":305}," is fine; restructuring the ",{"type":55,"tag":83,"props":307,"children":308},{},[309],{"type":61,"value":310},"dataflow",{"type":61,"value":312}," is not.\nIf the script cannot be made to run after non-destructive setup (missing\ndependencies, broken code), report the blocking issue as the deliverable\nand stop — do not proceed to verify a program that never ran.",{"type":55,"tag":159,"props":314,"children":315},{},[316,321,323,329,331],{"type":55,"tag":75,"props":317,"children":318},{},[319],{"type":61,"value":320},"Instrument and establish ground truth",{"type":61,"value":322}," (details and templates in\n",{"type":55,"tag":103,"props":324,"children":326},{"className":325},[],[327],{"type":61,"value":328},"references\u002Fverification.md",{"type":61,"value":330},"):",{"type":55,"tag":155,"props":332,"children":333},{},[334,347],{"type":55,"tag":159,"props":335,"children":336},{},[337,339,345],{"type":61,"value":338},"Set ",{"type":55,"tag":103,"props":340,"children":342},{"className":341},[],[343],{"type":61,"value":344},"wp.config.verify_autograd_array_access = True",{"type":61,"value":346}," before module load\nand rerun under an active tape. Capture every warning. This catches the\nsingle most common bug class (write-after-read overwrites) nearly for\nfree. Know its blind spots: it needs a tape, it cannot see arrays stored\ninside Warp structs, and it disables kernel caching (expect a kernel\nrebuild — JIT module recompilation only, not a rebuild of the native\nlibrary). If the tracker runs clean but gradients are still wrong,\nspecifically check for in-place mutations of arrays held inside Warp\nstructs (quick-checks §1 and Limitations) before trusting the clean\nresult.",{"type":55,"tag":159,"props":348,"children":349},{},[350,352,357,359,364,366,371,373,378,380,385,387,392,394,399],{"type":61,"value":351},"Run one ",{"type":55,"tag":75,"props":353,"children":354},{},[355],{"type":61,"value":356},"end-to-end finite-difference check",{"type":61,"value":358},": wrap the full forward\npass (sim steps + loss) in a Python callable and hand it to\n",{"type":55,"tag":103,"props":360,"children":362},{"className":361},[],[363],{"type":61,"value":116},{"type":61,"value":365}," with the true optimization inputs — it compares\nthe autodiff gradient against central differences, restoring array\ninputs between evaluations (Warp 1.17+) so in-place-mutating forwards\nare checked from pristine state; on older Warp use the manual harness\nin ",{"type":55,"tag":103,"props":367,"children":369},{"className":368},[],[370],{"type":61,"value":328},{"type":61,"value":372},". The reference is the user's ",{"type":55,"tag":83,"props":374,"children":375},{},[376],{"type":61,"value":377},"actual\nobjective over\nthe full horizon",{"type":61,"value":379},", compared against the gradient the optimizer ",{"type":55,"tag":83,"props":381,"children":382},{},[383],{"type":61,"value":384},"actually\nconsumes",{"type":61,"value":386}," — never a narrower window (see ",{"type":55,"tag":103,"props":388,"children":390},{"className":389},[],[391],{"type":61,"value":328},{"type":61,"value":393},").\nThis confirms gradients are actually wrong (users are sometimes wrong\nabout this — report \"gradients are correct\" findings honestly) and\nyields the error signature. The template in\n",{"type":55,"tag":103,"props":395,"children":397},{"className":396},[],[398],{"type":61,"value":328},{"type":61,"value":400}," fixes the eps\u002Ftolerance choices and the\nseed-pinning a stochastic forward needs — do not eyeball pass\u002Ffail\nagainst floating-point or sampling noise. If the backward pass runs out\nof memory while establishing ground truth, apply the checkpointing\npattern from \"Edge case: out of memory\" below before proceeding.",{"type":55,"tag":159,"props":402,"children":403},{},[404,409],{"type":55,"tag":75,"props":405,"children":406},{},[407],{"type":61,"value":408},"Match the signature",{"type":61,"value":410}," against the table below to rank hypotheses.",{"type":55,"tag":159,"props":412,"children":413},{},[414,419,421,427],{"type":55,"tag":75,"props":415,"children":416},{},[417],{"type":61,"value":418},"Scan the code against the known-pattern checklist",{"type":61,"value":420},"\n(",{"type":55,"tag":103,"props":422,"children":424},{"className":423},[],[425],{"type":61,"value":426},"references\u002Fquick-checks.md",{"type":61,"value":428},"). This is fast for you — do it in the same\npass, but let the signature decide which findings are plausible causes\nversus incidental smells.",{"type":55,"tag":159,"props":430,"children":431},{},[432,437,439,444,446,451,453,458,460,465,467,473],{"type":55,"tag":75,"props":433,"children":434},{},[435],{"type":61,"value":436},"Localize if still ambiguous.",{"type":61,"value":438}," Binary-search the pipeline: truncate to K\nsteps and find where FD and autodiff first diverge; run\n",{"type":55,"tag":103,"props":440,"children":442},{"className":441},[],[443],{"type":61,"value":205},{"type":61,"value":445}," to test each recorded launch in isolation.\nRemember gradcheck_tape validates kernels ",{"type":55,"tag":83,"props":447,"children":448},{},[449],{"type":61,"value":450},"individually",{"type":61,"value":452}," — it is\nstructurally blind to inter-kernel overwrites, so a clean per-kernel pass\nplus a wrong end-to-end gradient points ",{"type":55,"tag":83,"props":454,"children":455},{},[456],{"type":61,"value":457},"at",{"type":61,"value":459}," the taping pattern, not the\nkernels. It also silently ",{"type":55,"tag":83,"props":461,"children":462},{},[463],{"type":61,"value":464},"skips",{"type":61,"value":466}," kernels compiled with\n",{"type":55,"tag":103,"props":468,"children":470},{"className":469},[],[471],{"type":61,"value":472},"enable_backward=False",{"type":61,"value":474}," (see Limitations) — if any kernel in the pipeline\nsets that, a clean pass says nothing about it; verify it separately.",{"type":55,"tag":159,"props":476,"children":477},{},[478,483,485,490,492,497],{"type":55,"tag":75,"props":479,"children":480},{},[481],{"type":61,"value":482},"Fix minimally, then re-verify",{"type":61,"value":484}," with the exact same FD harness that\nestablished the failure. A gradient fix without a before\u002Fafter FD\ncomparison is not a fix. Verify the exact program you are shipping — the\nfixed file as it stands, every line included — never a re-implementation\nof it in a diagnostic script: a rebuilt pipeline silently drops whatever\nyou believed was irrelevant, and if that belief is wrong the verification\npasses while the shipped code stays broken. Mechanically: the harness\nmust ",{"type":55,"tag":83,"props":486,"children":487},{},[488],{"type":61,"value":489},"import the fixed module (or execute the fixed file) and call into\nit",{"type":61,"value":491}," — the only code that may live outside the shipped program is the FD\ndriver itself. Also rerun the overwrite\ntracker to confirm the warnings are gone. \"Minimally\" applies to the code\ndiff, not the diagnosis:\nwhen the root cause is structural (e.g., accidental gradient truncation,\nquick-checks §8), the minimal ",{"type":55,"tag":83,"props":493,"children":494},{},[495],{"type":61,"value":496},"correct",{"type":61,"value":498}," fix is the restructure — do not\nsubstitute a smaller change that only silences the surface symptom.",{"type":55,"tag":159,"props":500,"children":501},{},[502,507,509,514],{"type":55,"tag":75,"props":503,"children":504},{},[505],{"type":61,"value":506},"Close the loop on the user's original complaint.",{"type":61,"value":508}," Rerun their actual\nworkflow (their script, their printed metrics). The job is done when the\nsymptom they reported is resolved — an optimization that was \"exploding\"\nshould now demonstrably ",{"type":55,"tag":83,"props":510,"children":511},{},[512],{"type":61,"value":513},"improve its objective",{"type":61,"value":515},", not merely avoid NaN. If\ngradients verify correct at the full horizon but training still fails,\nthat is a new signature-table entry, not a victory; keep diagnosing (or\nreport the verified gradients and the remaining non-gradient cause, e.g.\nlearning rate).",{"type":55,"tag":91,"props":517,"children":519},{"id":518},"failure-signatures",[520],{"type":61,"value":521},"Failure signatures",{"type":55,"tag":523,"props":524,"children":525},"table",{},[526,545],{"type":55,"tag":527,"props":528,"children":529},"thead",{},[530],{"type":55,"tag":531,"props":532,"children":533},"tr",{},[534,540],{"type":55,"tag":535,"props":536,"children":537},"th",{},[538],{"type":61,"value":539},"Signature",{"type":55,"tag":535,"props":541,"children":542},{},[543],{"type":61,"value":544},"Leading hypotheses",{"type":55,"tag":546,"props":547,"children":548},"tbody",{},[549,653,679,714,779,821,847,860],{"type":55,"tag":531,"props":550,"children":551},{},[552,558],{"type":55,"tag":553,"props":554,"children":555},"td",{},[556],{"type":61,"value":557},"Gradients exactly zero",{"type":55,"tag":553,"props":559,"children":560},{},[561,563,569,571,577,579,585,587,593,595,601,603,608,610,616,618,624,625,631,632,638,640,644,646,651],{"type":61,"value":562},"Missing ",{"type":55,"tag":103,"props":564,"children":566},{"className":565},[],[567],{"type":61,"value":568},"requires_grad=True",{"type":61,"value":570}," somewhere in the chain (note ",{"type":55,"tag":103,"props":572,"children":574},{"className":573},[],[575],{"type":61,"value":576},"wp.zeros",{"type":61,"value":578}," defaults to ",{"type":55,"tag":103,"props":580,"children":582},{"className":581},[],[583],{"type":61,"value":584},"False",{"type":61,"value":586},"; ",{"type":55,"tag":103,"props":588,"children":590},{"className":589},[],[591],{"type":61,"value":592},"zeros_like",{"type":61,"value":594},"\u002F",{"type":55,"tag":103,"props":596,"children":598},{"className":597},[],[599],{"type":61,"value":600},"clone",{"type":61,"value":602}," inherit from source); ",{"type":55,"tag":103,"props":604,"children":606},{"className":605},[],[607],{"type":61,"value":472},{"type":61,"value":609}," at module\u002Fkernel level; loss array not connected to the tape; grads read after ",{"type":55,"tag":103,"props":611,"children":613},{"className":612},[],[614],{"type":61,"value":615},"tape.zero()",{"type":61,"value":617},"; a piecewise-constant op (",{"type":55,"tag":103,"props":619,"children":621},{"className":620},[],[622],{"type":61,"value":623},"round",{"type":61,"value":594},{"type":55,"tag":103,"props":626,"children":628},{"className":627},[],[629],{"type":61,"value":630},"floor",{"type":61,"value":594},{"type":55,"tag":103,"props":633,"children":635},{"className":634},[],[636],{"type":61,"value":637},"sign",{"type":61,"value":639},"\u002Fcast\u002Fthreshold) in the chain — there zero is ",{"type":55,"tag":83,"props":641,"children":642},{},[643],{"type":61,"value":496},{"type":61,"value":645}," and the fix is a surrogate gradient such as a straight-through estimator, not a bug hunt (quick-checks §9c); on Warp \u003C 1.17, a tape-recorded copy\u002Fclone whose source has other downstream readers (see the version caveat in ",{"type":55,"tag":103,"props":647,"children":649},{"className":648},[],[650],{"type":61,"value":426},{"type":61,"value":652},")",{"type":55,"tag":531,"props":654,"children":655},{},[656,661],{"type":55,"tag":553,"props":657,"children":658},{},[659],{"type":61,"value":660},"Gradients grow without bound across optimizer iterations",{"type":55,"tag":553,"props":662,"children":663},{},[664,665,670,671,677],{"type":61,"value":562},{"type":55,"tag":103,"props":666,"children":668},{"className":667},[],[669],{"type":61,"value":615},{"type":61,"value":594},{"type":55,"tag":103,"props":672,"children":674},{"className":673},[],[675],{"type":61,"value":676},"tape.reset()",{"type":61,"value":678}," between iterations; state-object aliasing that carries an in-tape overwrite across frames (case study 1)",{"type":55,"tag":531,"props":680,"children":681},{},[682,687],{"type":55,"tag":553,"props":683,"children":684},{},[685],{"type":61,"value":686},"Off by an exact small factor (2x, Nx)",{"type":55,"tag":553,"props":688,"children":689},{},[690,692,698,700,705,707,712],{"type":61,"value":691},"Double accumulation: a duplicate launch recorded on the tape — note that since Warp 1.13 the store adjoint consumes the output gradient on first use, so a bare duplicate is inert unless the rewritten array has ",{"type":55,"tag":103,"props":693,"children":695},{"className":694},[],[696],{"type":61,"value":697},"retain_grad=True",{"type":61,"value":699}," (quick-checks §7) or the Warp version is older; overlapping tape scopes taping the same work twice. Also: a backward seed that does not match the stated objective — seeding a per-element loss adjoint with ones backpropagates the ",{"type":55,"tag":83,"props":701,"children":702},{},[703],{"type":61,"value":704},"sum",{"type":61,"value":706},", exactly N× the ",{"type":55,"tag":83,"props":708,"children":709},{},[710],{"type":61,"value":711},"mean",{"type":61,"value":713}," objective's gradient",{"type":55,"tag":531,"props":715,"children":716},{},[717,722],{"type":55,"tag":553,"props":718,"children":719},{},[720],{"type":61,"value":721},"NaN or inf",{"type":55,"tag":553,"props":723,"children":724},{},[725,727,733,734,740,741,747,749,755,757,762,764,770,772,778],{"type":61,"value":726},"Non-differentiable point evaluated in the backward pass (",{"type":55,"tag":103,"props":728,"children":730},{"className":729},[],[731],{"type":61,"value":732},"wp.sqrt(0)",{"type":61,"value":199},{"type":55,"tag":103,"props":735,"children":737},{"className":736},[],[738],{"type":61,"value":739},"wp.length(0)",{"type":61,"value":199},{"type":55,"tag":103,"props":742,"children":744},{"className":743},[],[745],{"type":61,"value":746},"wp.normalize(0)",{"type":61,"value":748},", division) — needs a custom gradient (",{"type":55,"tag":103,"props":750,"children":752},{"className":751},[],[753],{"type":61,"value":754},"references\u002Fcustom-gradients.md",{"type":61,"value":756},") or, better, a stable reformulation; an overflow evaluated in the ",{"type":55,"tag":83,"props":758,"children":759},{},[760],{"type":61,"value":761},"unselected",{"type":61,"value":763}," branch of ",{"type":55,"tag":103,"props":765,"children":767},{"className":766},[],[768],{"type":61,"value":769},"wp.where",{"type":61,"value":771}," (a select, not a branch — quick-checks §9b); dynamic-loop local not recomputed during replay (documented to produce ",{"type":55,"tag":103,"props":773,"children":775},{"className":774},[],[776],{"type":61,"value":777},"inf",{"type":61,"value":652},{"type":55,"tag":531,"props":780,"children":781},{},[782,787],{"type":55,"tag":553,"props":783,"children":784},{},[785],{"type":61,"value":786},"Subtly wrong, often worse with more steps\u002Fiterations",{"type":55,"tag":553,"props":788,"children":789},{},[790,792,798,800,806,807,813,815,820],{"type":61,"value":791},"Write-after-read overwrite: ",{"type":55,"tag":103,"props":793,"children":795},{"className":794},[],[796],{"type":61,"value":797},"wp.copy",{"type":61,"value":799}," onto an already-read array, ping-pong buffers within one tape, Python rebinding that aliases two \"different\" states (case studies); in-place ",{"type":55,"tag":103,"props":801,"children":803},{"className":802},[],[804],{"type":61,"value":805},"*=",{"type":61,"value":594},{"type":55,"tag":103,"props":808,"children":810},{"className":809},[],[811],{"type":61,"value":812},"\u002F=",{"type":61,"value":814},"; vector\u002Fmatrix component reassignment; dynamic-loop intermediates; on Warp \u003C 1.17, a recorded copy\u002Fclone that is not the last consumer of its source (version caveat in ",{"type":55,"tag":103,"props":816,"children":818},{"className":817},[],[819],{"type":61,"value":426},{"type":61,"value":652},{"type":55,"tag":531,"props":822,"children":823},{},[824,829],{"type":55,"tag":553,"props":825,"children":826},{},[827],{"type":61,"value":828},"Per-window FD agrees but full-horizon FD disagrees; or gradients \"verified\" yet the optimizer stalls or worsens the loss",{"type":55,"tag":553,"props":830,"children":831},{},[832,837,839,845],{"type":55,"tag":75,"props":833,"children":834},{},[835],{"type":61,"value":836},"Accidental gradient truncation",{"type":61,"value":838},": a tape-per-step loop with backward inside it and state carried between tapes optimizes a different objective than the one being reported (see quick-checks §8). The structural fix is one tape over the whole horizon with ",{"type":55,"tag":103,"props":840,"children":842},{"className":841},[],[843],{"type":61,"value":844},"total_steps + 1",{"type":61,"value":846}," distinct state buffers. The solver-space analog: a partially converged iterative solve inside the tape makes FD and autodiff agree on the wrong program — converge it outside the tape and warm-start the taped iterations (quick-checks §8)",{"type":55,"tag":531,"props":848,"children":849},{},[850,855],{"type":55,"tag":553,"props":851,"children":852},{},[853],{"type":61,"value":854},"Gradients disagree (vs a reference implementation or run-to-run) only on a sparse, data-dependent subset; forward outputs match to float precision",{"type":55,"tag":553,"props":856,"children":857},{},[858],{"type":61,"value":859},"Under-determined forward choice at a non-smooth point (quick-checks §9): both answers can be valid subgradients, and FD cannot adjudicate at a kink. Check whether the discrete choice differs at exactly the mismatching elements before hunting corruption",{"type":55,"tag":531,"props":861,"children":862},{},[863,875],{"type":55,"tag":553,"props":864,"children":865},{},[866,868,873],{"type":61,"value":867},"FD and autodiff agree ",{"type":55,"tag":83,"props":869,"children":870},{},[871],{"type":61,"value":872},"at the full horizon",{"type":61,"value":874}," but optimization still fails",{"type":55,"tag":553,"props":876,"children":877},{},[878],{"type":61,"value":879},"Not a gradient bug. Say so. Look at learning rate, loss landscape, physics stability — and report the verified-correct gradients as the finding",{"type":55,"tag":91,"props":881,"children":883},{"id":882},"examples",[884],{"type":61,"value":885},"Examples",{"type":55,"tag":64,"props":887,"children":888},{},[889],{"type":61,"value":890},"A representative session, end to end. A user reports \"my cloth sim trains for\na while, then the loss creeps back up — tuning the learning rate doesn't\nhelp.\" No mention of gradients; the leap is made because the workflow\noptimizes through Warp kernels.",{"type":55,"tag":228,"props":892,"children":893},{},[894,899,917,929,942,955],{"type":55,"tag":159,"props":895,"children":896},{},[897],{"type":61,"value":898},"Their script runs 512 particles for 200 steps per iteration. Shrink to 16\nparticles, 10 steps, CPU — repro now runs in ~2 s and shows the same\ncreep.",{"type":55,"tag":159,"props":900,"children":901},{},[902,907,909,915],{"type":55,"tag":103,"props":903,"children":905},{"className":904},[],[906],{"type":61,"value":344},{"type":61,"value":908}," under the tape prints:\n",{"type":55,"tag":103,"props":910,"children":912},{"className":911},[],[913],{"type":61,"value":914},"array ... was read from kernel integrate and is now being written to by kernel integrate",{"type":61,"value":916}," — a write-after-read overwrite.",{"type":55,"tag":159,"props":918,"children":919},{},[920,922,927],{"type":61,"value":921},"End-to-end ",{"type":55,"tag":103,"props":923,"children":925},{"className":924},[],[926],{"type":61,"value":116},{"type":61,"value":928}," on the shrunk repro: max relative error\n0.4 against finite differences. Gradients are confirmed wrong, with the\n\"subtly wrong, worse with more steps\" signature.",{"type":55,"tag":159,"props":930,"children":931},{},[932,934,940],{"type":61,"value":933},"The signature row plus quick-checks §1 point at buffer reuse inside one\ntape: the sim steps ",{"type":55,"tag":103,"props":935,"children":937},{"className":936},[],[938],{"type":61,"value":939},"state_a → state_b → state_a",{"type":61,"value":941},", ping-ponging two\nbuffers, so the backward pass reads clobbered states.",{"type":55,"tag":159,"props":943,"children":944},{},[945,947,953],{"type":61,"value":946},"Minimal fix: allocate ",{"type":55,"tag":103,"props":948,"children":950},{"className":949},[],[951],{"type":61,"value":952},"num_steps + 1",{"type":61,"value":954}," distinct state buffers recorded on\nthe tape (physics untouched; only the dataflow changes).",{"type":55,"tag":159,"props":956,"children":957},{},[958],{"type":61,"value":959},"Re-verify: same gradcheck harness now passes (max relative error 3e-4);\nthe overwrite warning is gone; the user's full-size training run now\ndecreases monotonically.",{"type":55,"tag":64,"props":961,"children":962},{},[963],{"type":61,"value":964},"Report: root cause (in-tape buffer reuse), the evidence chain (warning +\nbefore\u002Fafter FD numbers), the two-line diff, and a pointer to the\n\"Array Overwrites\" section of the Differentiability guide.",{"type":55,"tag":91,"props":966,"children":968},{"id":967},"reporting",[969],{"type":61,"value":970},"Reporting",{"type":55,"tag":64,"props":972,"children":973},{},[974],{"type":61,"value":975},"Lead with the root cause and the evidence chain: the FD-vs-autodiff numbers\nthat established the failure, the warning or localization step that found the\ncause, the minimal diff, and the FD numbers after the fix. Name the\ndocumentation section that covers the pattern so the user can read the\ncanonical explanation. If you checked patterns that came up clean (e.g., the\noverwrite tracker found nothing), say so — it tells the user what has been\nruled out.",{"type":55,"tag":64,"props":977,"children":978},{},[979,981,986],{"type":61,"value":980},"If the user is only asking ",{"type":55,"tag":83,"props":982,"children":983},{},[984],{"type":61,"value":985},"whether",{"type":61,"value":987}," their gradients are trustworthy, stop\nafter verification and report; apply fixes when they ask for fixes.",{"type":55,"tag":64,"props":989,"children":990},{},[991],{"type":61,"value":992},"Preserve the evidence: leave the diagnostic scripts (FD harness, shrunk\nrepro) in the workspace and list them in the report instead of deleting\nthem — they are the reproducible half of the evidence chain, and the user\nor a reviewer should be able to rerun the exact verification that\njustified the fix. Never delete files you did not create.",{"type":55,"tag":91,"props":994,"children":996},{"id":995},"limitations",[997],{"type":61,"value":998},"Limitations",{"type":55,"tag":64,"props":1000,"children":1001},{},[1002,1004,1009],{"type":61,"value":1003},"The verification tooling has blind spots — a clean pass through any one\ntool is not a clean bill of health (details in\n",{"type":55,"tag":103,"props":1005,"children":1007},{"className":1006},[],[1008],{"type":61,"value":328},{"type":61,"value":330},{"type":55,"tag":155,"props":1011,"children":1012},{},[1013,1018,1028,1045,1071,1076],{"type":55,"tag":159,"props":1014,"children":1015},{},[1016],{"type":61,"value":1017},"The overwrite tracker requires an active tape, cannot see arrays stored\ninside Warp structs, and disables kernel caching while enabled.",{"type":55,"tag":159,"props":1019,"children":1020},{},[1021,1026],{"type":55,"tag":103,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":61,"value":116},{"type":61,"value":1027}," does not accept struct inputs; wrap the forward\nin a callable over the underlying arrays. On Warp \u003C 1.17 it does not\nrestore mutated array inputs between evaluations (use the manual\nharness).",{"type":55,"tag":159,"props":1029,"children":1030},{},[1031,1036,1038,1043],{"type":55,"tag":103,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":61,"value":205},{"type":61,"value":1037}," validates each recorded launch in\nisolation — it is structurally blind to inter-kernel overwrite bugs and\nsilently skips kernels compiled with ",{"type":55,"tag":103,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":61,"value":472},{"type":61,"value":1044},".",{"type":55,"tag":159,"props":1046,"children":1047},{},[1048,1050,1055,1056,1061,1063,1069],{"type":61,"value":1049},"The ",{"type":55,"tag":103,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":61,"value":805},{"type":61,"value":594},{"type":55,"tag":103,"props":1057,"children":1059},{"className":1058},[],[1060],{"type":61,"value":812},{"type":61,"value":1062}," non-differentiability warning is emitted only at codegen\ntime under ",{"type":55,"tag":103,"props":1064,"children":1066},{"className":1065},[],[1067],{"type":61,"value":1068},"wp.LOG_DEBUG",{"type":61,"value":1070},", so its absence from a normal run means\nnothing.",{"type":55,"tag":159,"props":1072,"children":1073},{},[1074],{"type":61,"value":1075},"Warp has no built-in gradient checkpointing; long-horizon memory\npressure needs the application-level pattern below.",{"type":55,"tag":159,"props":1077,"children":1078},{},[1079],{"type":61,"value":1080},"At non-smooth points (ties, kinks, argmin selections), finite\ndifferences cannot adjudicate between valid subgradients — FD-vs-AD\ndisagreement there is not automatically a bug (quick-checks §9).",{"type":55,"tag":91,"props":1082,"children":1084},{"id":1083},"edge-case-out-of-memory",[1085],{"type":61,"value":1086},"Edge case: out of memory",{"type":55,"tag":64,"props":1088,"children":1089},{},[1090,1092,1098],{"type":61,"value":1091},"If the backward pass fails to allocate (long simulations keep every\nintermediate state alive on the tape), the fix is gradient checkpointing:\nsave periodic states, replay the segments between them during backward. Warp\nhas no built-in utility — applications implement it themselves. Use\n",{"type":55,"tag":103,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":61,"value":1097},"warp\u002Fexamples\u002Foptim\u002Fexample_fluid_checkpoint.py",{"type":61,"value":1099}," as the reference pattern,\nand see the FAQ's \"Differentiation and Interoperability\" section.",{"type":55,"tag":91,"props":1101,"children":1103},{"id":1102},"reference-files",[1104],{"type":61,"value":1105},"Reference files",{"type":55,"tag":155,"props":1107,"children":1108},{},[1109,1119,1137,1170],{"type":55,"tag":159,"props":1110,"children":1111},{},[1112,1117],{"type":55,"tag":103,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":61,"value":426},{"type":61,"value":1118}," — the known-bug-pattern checklist with doc\npointers and the caveats that make each pattern easy to miss.",{"type":55,"tag":159,"props":1120,"children":1121},{},[1122,1127,1129,1135],{"type":55,"tag":103,"props":1123,"children":1125},{"className":1124},[],[1126],{"type":61,"value":328},{"type":61,"value":1128}," — tooling details: overwrite tracker setup and\nblind spots, end-to-end FD harness template, ",{"type":55,"tag":103,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":61,"value":1134},"wp.autograd",{"type":61,"value":1136},"\ngradcheck\u002Fjacobian usage and caveats, tape visualization, bisection.",{"type":55,"tag":159,"props":1138,"children":1139},{},[1140,1145,1147,1153,1154,1160,1162,1168],{"type":55,"tag":103,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":61,"value":754},{"type":61,"value":1146}," — ",{"type":55,"tag":103,"props":1148,"children":1150},{"className":1149},[],[1151],{"type":61,"value":1152},"@wp.func_grad",{"type":61,"value":199},{"type":55,"tag":103,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":61,"value":1159},"@wp.func_replay",{"type":61,"value":1161},",\n",{"type":55,"tag":103,"props":1163,"children":1165},{"className":1164},[],[1166],{"type":61,"value":1167},"@wp.func_native",{"type":61,"value":1169},": when they are required and how they are misused.",{"type":55,"tag":159,"props":1171,"children":1172},{},[1173,1179],{"type":55,"tag":103,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":61,"value":1178},"references\u002Fcase-studies.md",{"type":61,"value":1180}," — two real debugging sagas (state aliasing;\ndifferentiable-copy overwrite) showing how subtle the surface symptoms are.\nRead these when the checklist comes up clean — they calibrate what \"subtle\"\nmeans here.",{"items":1182,"total":1213},[1183,1193,1200],{"slug":1184,"name":1184,"fn":1185,"description":1186,"org":1187,"tags":1188,"stars":23,"repoUrl":24,"updatedAt":1192},"warp-compile-time-optimizer","optimize Warp compile and startup times","Use when compile time or startup time is the problem in code that uses Warp: a request to improve, optimize, or cut compile times; an app that is slow to start or stalls at the first wp.launch; seconds of compiling before real work begins; JIT modules recompiling on every run or every CI job. Only applies when the code being optimized uses Warp kernels. Not for steady-state kernel runtime, memory, correctness, building Warp itself from source, or nvcc\u002FC++ build times.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1189,1190,1191],{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-08-05T05:58:13.178762",{"slug":4,"name":4,"fn":5,"description":6,"org":1194,"tags":1195,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1196,1197,1198,1199],{"name":21,"slug":22,"type":15},{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":1201,"name":1201,"fn":1202,"description":1203,"org":1204,"tags":1205,"stars":23,"repoUrl":24,"updatedAt":1212},"warp-eval","evaluate NVIDIA Warp simulation candidates","Evaluate whether an existing hot path is a credible NVIDIA Warp candidate. Use for irregular or spatial queries, particle or geometry simulation, branch-heavy loops, many small launches, host fallbacks, or large intermediates. CPU-only code and absent GPU dependencies are normal unless NVIDIA is prohibited. Exclude required cross-vendor or CPU-only deployment, vendor-lowered dense or NN layers, general Warp API questions, and already-selected Warp kernels. Contribution policy alone is not exclusion.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1206,1207,1208,1209],{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"name":1210,"slug":1211,"type":15},"Simulation","simulation","2026-08-05T05:58:27.275636",3,{"items":1215,"total":1371},[1216,1234,1251,1262,1274,1286,1299,1313,1326,1337,1351,1360],{"slug":1217,"name":1217,"fn":1218,"description":1219,"org":1220,"tags":1221,"stars":1231,"repoUrl":1232,"updatedAt":1233},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1222,1225,1228],{"name":1223,"slug":1224,"type":15},"Documentation","documentation",{"name":1226,"slug":1227,"type":15},"MCP","mcp",{"name":1229,"slug":1230,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1235,"name":1235,"fn":1236,"description":1237,"org":1238,"tags":1239,"stars":1248,"repoUrl":1249,"updatedAt":1250},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1240,1243,1246],{"name":1241,"slug":1242,"type":15},"Containers","containers",{"name":1244,"slug":1245,"type":15},"Deployment","deployment",{"name":1247,"slug":34,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1252,"name":1252,"fn":1253,"description":1254,"org":1255,"tags":1256,"stars":1248,"repoUrl":1249,"updatedAt":1261},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1257,1260],{"name":1258,"slug":1259,"type":15},"CI\u002FCD","ci-cd",{"name":1244,"slug":1245,"type":15},"2026-07-14T05:25:59.97109",{"slug":1263,"name":1263,"fn":1264,"description":1265,"org":1266,"tags":1267,"stars":1248,"repoUrl":1249,"updatedAt":1273},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1268,1269,1270],{"name":1258,"slug":1259,"type":15},{"name":1244,"slug":1245,"type":15},{"name":1271,"slug":1272,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1275,"name":1275,"fn":1276,"description":1277,"org":1278,"tags":1279,"stars":1248,"repoUrl":1249,"updatedAt":1285},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1280,1281,1282],{"name":21,"slug":22,"type":15},{"name":1271,"slug":1272,"type":15},{"name":1283,"slug":1284,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1287,"name":1287,"fn":1288,"description":1289,"org":1290,"tags":1291,"stars":1248,"repoUrl":1249,"updatedAt":1298},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1292,1295],{"name":1293,"slug":1294,"type":15},"Best Practices","best-practices",{"name":1296,"slug":1297,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1300,"name":1300,"fn":1301,"description":1302,"org":1303,"tags":1304,"stars":1248,"repoUrl":1249,"updatedAt":1312},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1305,1308,1311],{"name":1306,"slug":1307,"type":15},"Machine Learning","machine-learning",{"name":1309,"slug":1310,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1314,"name":1314,"fn":1315,"description":1316,"org":1317,"tags":1318,"stars":1248,"repoUrl":1249,"updatedAt":1325},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1319,1322],{"name":1320,"slug":1321,"type":15},"QA","qa",{"name":1323,"slug":1324,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1327,"name":1327,"fn":1328,"description":1329,"org":1330,"tags":1331,"stars":1248,"repoUrl":1249,"updatedAt":1336},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1332,1333],{"name":1244,"slug":1245,"type":15},{"name":1334,"slug":1335,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1338,"name":1338,"fn":1339,"description":1340,"org":1341,"tags":1342,"stars":1248,"repoUrl":1249,"updatedAt":1350},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1343,1346,1347],{"name":1344,"slug":1345,"type":15},"Code Review","code-review",{"name":1271,"slug":1272,"type":15},{"name":1348,"slug":1349,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1352,"name":1352,"fn":1353,"description":1354,"org":1355,"tags":1356,"stars":1248,"repoUrl":1249,"updatedAt":1359},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1357,1358],{"name":1320,"slug":1321,"type":15},{"name":1323,"slug":1324,"type":15},"2026-07-14T05:25:54.928983",{"slug":1361,"name":1361,"fn":1362,"description":1363,"org":1364,"tags":1365,"stars":1248,"repoUrl":1249,"updatedAt":1370},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1366,1369],{"name":1367,"slug":1368,"type":15},"Automation","automation",{"name":1258,"slug":1259,"type":15},"2026-07-30T05:29:03.275638",525]