[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-cuda-kernels":3,"mdc--otw1cm-key":38,"related-repo-huggingface-cuda-kernels":5902,"related-org-huggingface-cuda-kernels":5954},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"cuda-kernels","write and benchmark optimized CUDA kernels","Provides guidance for writing and benchmarking optimized CUDA kernels for NVIDIA GPUs (H100, A100, T4) targeting HuggingFace diffusers and transformers libraries. Kernels must be kernel-builder\u002FABI3-compliant: no pybind11, no setup.py, TORCH_LIBRARY_EXPAND bindings only. Supports models like LTX-Video, Stable Diffusion, LLaMA, Mistral, and Qwen. Includes integration with HuggingFace Kernels Hub (get_kernel) for loading pre-compiled kernels. Includes benchmarking scripts to compare kernel performance against baseline implementations.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,16,18,21,24],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":17,"type":15},"hugging-face",{"name":19,"slug":20,"type":15},"Deep Learning","deep-learning",{"name":22,"slug":23,"type":15},"Python","python",{"name":25,"slug":26,"type":15},"AI Infrastructure","ai-infrastructure",712,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels","2026-05-15T06:18:29.266429",null,112,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Build compute kernels and load them from the Hub.","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002FHEAD\u002Fkernel-builder\u002Fskills\u002Fcuda-kernels","---\nname: cuda-kernels\ndescription: \"Provides guidance for writing and benchmarking optimized CUDA kernels for NVIDIA GPUs (H100, A100, T4) targeting HuggingFace diffusers and transformers libraries. Kernels must be kernel-builder\u002FABI3-compliant: no pybind11, no setup.py, TORCH_LIBRARY_EXPAND bindings only. Supports models like LTX-Video, Stable Diffusion, LLaMA, Mistral, and Qwen. Includes integration with HuggingFace Kernels Hub (get_kernel) for loading pre-compiled kernels. Includes benchmarking scripts to compare kernel performance against baseline implementations.\"\ndisable-model-invocation: false\nuser-invocable: true\nallowed-tools: \"Read, Grep, Glob, Bash\"\nargument-hint: \"kernel type: attention, rmsnorm, rope, adaln, geglu, benchmark, transformers, diffusers, huggingface-kernels, get_kernel\"\n---\n\n# CUDA Kernels for Diffusers & Transformers\n\nThis skill provides patterns and guidance for developing optimized CUDA kernels targeting NVIDIA GPUs (H100, A100, T4) for use with HuggingFace **diffusers** and **transformers** libraries.\n\n## Hard Constraints — Read Before Writing Any Code\n\nKernels MUST build with [kernel-builder](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels) and meet the [Kernel Hub requirements](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fkernels\u002Fkernel-requirements). kernel-builder compiles against the **Python limited API (ABI3)** so a single binary works for Python 3.9+ across versions. Several patterns that are standard in generic PyTorch-extension tutorials are therefore **hard build failures** here. Do not use them, even if PyTorch documentation or your training data suggests them.\n\n### Disallowed patterns — never generate these\n\n| ❌ Never use | Why it fails | ✅ Use instead |\n|---|---|---|\n| pybind11 in any form: `#include \u003Ctorch\u002Fextension.h>`, `#include \u003Cpybind11\u002F...>`, `PYBIND11_MODULE(...)`, `py::arg`, any `py::` symbol | pybind11 is incompatible with the limited API (ABI3); the build does not compile | `TORCH_LIBRARY_EXPAND` in `torch-ext\u002Ftorch_binding.cpp` (see below). Note: `torch\u002Fextension.h` transitively includes pybind11 — include `torch\u002Ftorch.h` + `torch\u002Flibrary.h` instead |\n| Hand-written `setup.py` \u002F `pyproject.toml` using `torch.utils.cpp_extension` (`CUDAExtension`, `BuildExtension`, `cpp_extension.load`, `load_inline`) | setuptools extensions are not ABI3 and bypass `build.toml`; kernel-builder owns the build | `build.toml` + `nix run .#build-and-copy -L`. For an editable dev install, generate the project files with `kernel-builder create-pyproject -f` — never write them by hand |\n| `TORCH_LIBRARY(my_kernel, m)`, `TORCH_LIBRARY_FRAGMENT(...)`, or `TORCH_LIBRARY_IMPL(...)` with a hardcoded namespace | kernel-builder suffixes the op namespace with a per-build hash (e.g. `_my_kernel_a1b2c3d`); a hardcoded name never resolves | `TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops)` from the generated `registration.h` |\n| Hardcoded `torch.ops.my_kernel.fn(...)` calls in Python | Same namespace mangling — the op namespace name is only known at build time | `from ._ops import ops` then `ops.fn(...)` |\n| Hand-written `PyMODINIT_FUNC PyInit__...` or any manual CPython module init | Generated by `REGISTER_EXTENSION`; duplicating it breaks module loading | `REGISTER_EXTENSION(TORCH_EXTENSION_NAME)` exactly once, in `torch_binding.cpp` |\n| Non-limited CPython API calls (`PyArg_ParseTuple`, direct `PyObject*` manipulation) | Violates ABI3 | Stay within the torch C++ API: `torch::Tensor`, `TORCH_CHECK`, `at::cuda::*` |\n| Absolute imports of your own package inside `torch-ext\u002F` (`from my_kernel.utils import x`) | The package directory is renamed when loaded from the Hub; absolute imports break | Relative imports only: `from .utils import x`, `from ._ops import ops` |\n| Runtime Python deps beyond `torch` (and `einops` if truly needed) | Hub compliance restricts kernel dependencies; imports of numpy, triton, packaging, etc. are rejected | Standard library + `torch` only |\n| Python-side `@torch.library.custom_op` as the primary binding | The op must be registered in C++ so it ships in the compiled extension | C++ registration via `TORCH_LIBRARY_EXPAND`; Python-side `torch.library.register_fake` is only for adding a fake\u002Fmeta impl (see torch.compile section) |\n\n### The only supported binding pattern\n\n`registration.h` and `_ops.py` are **generated by kernel-builder** — reference them, never write them yourself.\n\n**`torch-ext\u002Ftorch_binding.h`:**\n```cpp\n#pragma once\n#include \u003Ctorch\u002Ftorch.h>\n\nvoid my_kernel_forward(torch::Tensor &out, torch::Tensor const &input);\n```\n\n**`torch-ext\u002Ftorch_binding.cpp`:**\n```cpp\n#include \u003Ctorch\u002Flibrary.h>\n#include \"registration.h\"\n#include \"torch_binding.h\"\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n  ops.def(\"my_kernel_forward(Tensor! out, Tensor input) -> ()\");\n  ops.impl(\"my_kernel_forward\", torch::kCUDA, &my_kernel_forward);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n```\n\n**`torch-ext\u002Fmy_kernel\u002F__init__.py`:**\n```python\nimport torch\nfrom ._ops import ops\n\ndef my_kernel(x: torch.Tensor) -> torch.Tensor:\n    out = torch.empty_like(x)\n    ops.my_kernel_forward(out, x)\n    return out\n```\n\n### Pre-flight checklist before declaring a kernel done\n\n1. `grep -rn \"pybind11\\|PYBIND11\\|torch\u002Fextension.h\\|py::\" torch-ext\u002F` returns nothing.\n2. `grep -rn \"TORCH_LIBRARY(\\|TORCH_LIBRARY_FRAGMENT\\|PyInit\" torch-ext\u002F` returns nothing (only `TORCH_LIBRARY_EXPAND` is allowed).\n3. No `setup.py` exists unless generated by `kernel-builder create-pyproject`.\n4. `kernel-builder check-config` passes — `[general]` needs a **dash-separated** `name` (never underscores) and a `license`, plus `[torch]` (binding sources) and `[kernel.\u003Cname>]` sections.\n5. The kernel directory is a git repository with all files committed (Nix refuses non-git builds).\n6. The build succeeds: `nix run .#build-and-copy -L`.\n7. ABI compliance passes: `kernel-builder check-abi` (after building).\n\n## Quick Start\n\n### Diffusers (Video\u002FImage Generation)\n\n**For benchmarking kernel performance:**\n```bash\n# Benchmark with optimized kernels (6% end-to-end speedup)\npython generate_video.py --use-optimized-kernels\n\n# Benchmark baseline with torch.compile (34% speedup)\npython generate_video.py --no-optimized-kernels --compile\n\n# Compare configurations (note: --compile and --use-optimized-kernels are mutually exclusive)\npython generate_video.py --use-optimized-kernels && \\\npython generate_video.py --no-optimized-kernels --compile\n```\n\n**For a minimal diffusers integration example (~150 lines):**\n```bash\npython scripts\u002Fltx_kernel_injection_example.py\n```\n\n### Transformers (LLMs)\n\n**For a minimal transformers integration example (~120 lines):**\n```bash\npython scripts\u002Ftransformers_injection_example.py\n```\n\n### HuggingFace Kernels Hub\n\n**Load pre-compiled kernels from HuggingFace Hub (no local compilation):**\n```python\nfrom kernels import get_kernel\n\n# Load optimized activation kernels\nactivation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n\n# Use the kernel\ny = torch.empty_like(x)\nactivation.gelu_fast(y, x)\n```\n\n**For a complete HuggingFace Kernels example:**\n```bash\npython scripts\u002Fhuggingface_kernels_example.py\n```\n\n### Isolated Kernel Micro-benchmarks\n\n```bash\npython benchmark_rmsnorm.py\n```\n\n## Supported Libraries & Models\n\n| Library | Supported Models | Key Kernels |\n|---------|------------------|-------------|\n| **diffusers** | LTX-Video, Stable Diffusion, FLUX, DiT | RMSNorm, GEGLU, RoPE, AdaLN |\n| **transformers** | LLaMA, Mistral, Qwen, Falcon | RMSNorm, Attention |\n\n| GPU | Compute Capability | Guide |\n|-----|-------------------|-------|\n| H100 | sm_90 | [h100-optimization-guide.md](references\u002Fh100-optimization-guide.md) |\n| A100 | sm_80 | [a100-optimization-guide.md](references\u002Fa100-optimization-guide.md) |\n| T4 | sm_75 | [t4-optimization-guide.md](references\u002Ft4-optimization-guide.md) |\n\n## When This Skill Applies\n\nUse this skill when:\n- **Benchmarking kernel performance** against baseline implementations\n- Writing new CUDA kernels for diffusion models or LLMs\n- Optimizing existing kernels for H100, A100, or T4 architecture\n- Implementing custom attention, normalization, or activation layers\n- Integrating kernels with **diffusers** pipelines (LTX-Video, Stable Diffusion, FLUX, DiT)\n- Integrating kernels with **transformers** models (LLaMA, Mistral, Qwen)\n- Debugging kernel performance issues on NVIDIA GPUs\n\n## Working Example\n\nComplete working examples ship with the kernels repo under `examples\u002Fkernels\u002F` (also at [github.com\u002Fhuggingface\u002Fkernels](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002Fmain\u002Fexamples\u002Fkernels)):\n- `relu\u002F` — the canonical minimal kernel: build.toml, flake.nix, `TORCH_LIBRARY_EXPAND` bindings, Python API, `layers\u002F`, tests\n- `relu-backprop-compile\u002F` — backward pass + `torch.compile` support (fake-op registration)\n- `silu-and-mul\u002F` — activation kernel following the same layout\n\n## Benchmarking Kernels\n\nUse the benchmark script to measure kernel performance:\n\n```bash\n# Full benchmark with all options\npython scripts\u002Fbenchmark_example.py \\\n    --use-optimized-kernels \\\n    --compile \\\n    --batch-size 1 \\\n    --num-frames 161 \\\n    --height 512 \\\n    --width 768 \\\n    --steps 50 \\\n    --warmup-iterations 2\n```\n\n### Benchmark Script Options\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--use-optimized-kernels` | auto | Use custom H100 CUDA kernels |\n| `--no-optimized-kernels` | - | Use baseline implementation |\n| `--compile` | false | Enable torch.compile on transformer |\n| `--batch-size` | 1 | Number of videos per prompt |\n| `--num-frames` | 161 | Number of frames to generate |\n| `--height` | 512 | Video height in pixels |\n| `--width` | 768 | Video width in pixels |\n| `--steps` | 50 | Denoising steps |\n| `--warmup-iterations` | 2 | Warmup runs before benchmark |\n\n### Example Benchmark Results\n\n**End-to-End Video Generation (49 frames, 30 steps, H100 80GB):**\n\n| Configuration | Time (s) | it\u002Fs | Speedup | Notes |\n|:---|:---:|:---:|:---:|:---|\n| Baseline (no compile) | 2.87 | 12.58 | 1.00x | Reference |\n| **Optimized Kernels** | 2.70 | 13.52 | **1.06x** | 6% faster |\n| Baseline + torch.compile | 2.14 | 19.05 | 1.34x | 34% faster |\n\n**Important:** `--use-optimized-kernels` and `--compile` are currently mutually exclusive. Custom kernels require PyTorch custom op registration to work with torch.compile.\n\n**Key metrics to capture:**\n- **Device:** GPU model (e.g., NVIDIA H100 80GB HBM3)\n- **Precision:** Data type used (e.g., bfloat16)\n- **Resolution:** Width x Height (e.g., 768x512)\n- **Frames:** Number of frames generated (e.g., 49, 161)\n\n### RMSNorm Micro-benchmarks\n\nThe vectorized RMSNorm kernel achieves **2.67x average speedup** over PyTorch baseline:\n\n| Shape | Custom (ms) | PyTorch (ms) | Speedup |\n|:---|:---:|:---:|:---:|\n| [1×1024×2048] | 0.019 | 0.065 | **3.37x** |\n| [2×1024×2048] | 0.024 | 0.073 | **3.04x** |\n| [4×1024×2048] | 0.036 | 0.093 | **2.58x** |\n| [2×4096×3072] | 0.087 | 0.208 | **2.41x** |\n| [4×4096×3072] | 0.157 | 0.392 | **2.49x** |\n\n**Bandwidth efficiency:** 38% of H100's theoretical 3.35 TB\u002Fs\n\n**Why end-to-end speedup is smaller:** RMSNorm accounts for ~5% of total compute in LTX-Video. The remaining time is spent in attention (Flash Attention\u002FSDPA), linear projections, and VAE decode.\n\n## Project Structure\n\n```\n.claude\u002Fskills\u002Fcuda-kernels\u002F\n├── scripts\u002F\n│   ├── benchmark_example.py              # End-to-end video generation benchmark\n│   ├── benchmark_rmsnorm.py              # Isolated RMSNorm micro-benchmark\n│   ├── ltx_kernel_injection_example.py   # Minimal diffusers integration (~150 lines)\n│   ├── transformers_injection_example.py # Minimal transformers integration (~120 lines)\n│   └── huggingface_kernels_example.py    # HuggingFace Kernels Hub integration\n├── references\u002F\n│   ├── diffusers-integration.md          # Complete diffusers integration guide\n│   ├── transformers-integration.md       # Complete transformers integration guide\n│   ├── huggingface-kernels-integration.md # HuggingFace Kernels Hub (get_kernel) guide\n│   ├── troubleshooting.md                # Common issues and solutions\n│   ├── kernel-templates.md               # CUDA kernel templates (includes vectorized)\n│   ├── h100-optimization-guide.md        # H100 (Hopper) optimization deep dive\n│   ├── a100-optimization-guide.md        # A100 (Ampere) optimization deep dive\n│   └── t4-optimization-guide.md          # T4 (Turing) optimization deep dive\n└── SKILL.md                              # This file\n\nexamples\u002Fkernels\u002Frelu\u002F               # Canonical working example (kernels repo)\n├── build.toml                      # kernel-builder build configuration\n├── flake.nix                       # Nix build entry point\n├── CARD.md                         # Kernel card template (becomes README.md)\n├── relu_cuda\u002Frelu.cu               # CUDA kernel source\n├── torch-ext\u002F\n│   ├── torch_binding.h \u002F .cpp      # TORCH_LIBRARY_EXPAND bindings\n│   └── relu\u002F__init__.py            # Python API (+ optional layers\u002F)\n└── tests\u002Ftest_relu.py              # Kernel tests (nix run .#ci-test)\n```\n\n## GPU Architecture Reference\n\n### H100 (Hopper) - Primary Target\n\n| Spec | Value | Optimization Impact |\n|------|-------|---------------------|\n| SMs | 132 | Grid sizing: aim for multiples of 132 |\n| Threads\u002FSM | 2048 | Max 16 blocks of 128 threads per SM |\n| Shared Memory | 192 KB\u002FSM | Large tiles possible |\n| L2 Cache | 50 MB | Reuse across blocks |\n| Memory BW | 3.35 TB\u002Fs | Coalesced access critical |\n| Warp Size | 32 | All reductions use warp shuffles |\n\n### Quick Comparison (H100 vs A100 vs T4)\n\n| Spec | H100 | A100 | T4 |\n|------|------|------|-----|\n| SMs | 132 | 108 | 40 |\n| Memory BW | 3.35 TB\u002Fs | 2.0 TB\u002Fs | 320 GB\u002Fs |\n| Shared Mem\u002FSM | 192 KB | 164 KB | 64 KB |\n| BF16 Support | Yes | Yes | **No (FP16 only)** |\n| Compute Cap | sm_90 | sm_80 | sm_75 |\n\n> See detailed guides: [H100](references\u002Fh100-optimization-guide.md) | [A100](references\u002Fa100-optimization-guide.md) | [T4](references\u002Ft4-optimization-guide.md)\n\n## Core Kernel Patterns\n\n### Vectorized Memory Access (Critical for Performance)\n\n**BFloat16 vectorization using `__nv_bfloat162`:**\n```cuda\n\u002F\u002F Load 2 bfloat16 elements at once (32-bit load)\nconst __nv_bfloat162* vec_input = reinterpret_cast\u003Cconst __nv_bfloat162*>(row_input);\n\n#pragma unroll 4\nfor (int i = tid; i \u003C vec_hidden; i += stride) {\n    __nv_bfloat162 v = vec_input[i];\n    float v0 = __bfloat162float(v.x);\n    float v1 = __bfloat162float(v.y);\n    sum_sq += v0 * v0 + v1 * v1;\n}\n```\n\n**FP16 vectorization using `__half2`:**\n```cuda\nconst __half2* vec_input = reinterpret_cast\u003Cconst __half2*>(row_input);\n__half2 v = vec_input[i];\nfloat v0 = __half2float(v.x);\nfloat v1 = __half2float(v.y);\n```\n\n**FP32 vectorization using `float4`:**\n```cuda\nconst float4* vec_input = reinterpret_cast\u003Cconst float4*>(row_input);\nfloat4 v = vec_input[i];\nsum_sq += v.x * v.x + v.y * v.y + v.z * v.z + v.w * v.w;\n```\n\n### Warp Shuffle Reductions\n```cuda\ntemplate \u003Ctypename T>\n__device__ __forceinline__ T warp_reduce_sum(T val) {\n    #pragma unroll\n    for (int offset = 16; offset > 0; offset >>= 1) {\n        val += __shfl_xor_sync(0xffffffff, val, offset);\n    }\n    return val;\n}\n```\n\n### Block Sizes for Attention\n- `BLOCK_SIZE_M = 128`, `BLOCK_SIZE_N = 64`, `BLOCK_SIZE_K = 64`\n- `NUM_WARPS = 8`\n\n### Thread Configuration\n\nFor element-wise ops (RoPE, GEGLU):\n```cuda\nconstexpr int BLOCK_SIZE = 256;\nint num_blocks = (total_elements + BLOCK_SIZE - 1) \u002F BLOCK_SIZE;\n```\n\nFor reduction ops (LayerNorm, RMSNorm) with vectorization:\n```cuda\n\u002F\u002F Divide by 2 for bf16\u002Ffp16 vectorized access\nint threads = min(hidden_size \u002F 2, MAX_THREADS);\nthreads = max(threads, WARP_SIZE);\nthreads = (threads + 32 - 1) \u002F 32 * 32;  \u002F\u002F Round to warp boundary\n```\n\n## Supported Data Types\n\nAll kernels support three precision modes:\n- `__half` (FP16) - Default for inference\n- `__nv_bfloat16` (BF16) - Preferred for training\n- `float` (FP32) - Reference\u002Fdebugging\n\n## Building Kernels\n\n### Scaffold a new kernel project\n\nStart new kernels with `kernel-builder init` instead of creating files by hand — it generates the compliant layout in one shot:\n\n```bash\nkernel-builder init --name my-username\u002Fmy-kernel\n```\n\nThis creates `build.toml` (valid dash-separated name, license, `[general.hub] repo-id` already wired), `flake.nix`, `torch-ext\u002F` with compilable `torch_binding.{h,cpp}` and the Python package, a `\u003Cname>_cuda\u002F` kernel source dir, `tests\u002F`, `benchmarks\u002F`, `example.py`, and `CARD.md` — and it initializes a git repository (required for builds). Then replace the stub kernel with your own sources and update the `src` lists in `build.toml`.\n\n### With Nix (Recommended)\n```bash\nnix run .#build-and-copy --max-jobs 2 --cores 8 -L\n```\n\n### Build and publish to the Hub in one go\n```bash\nkernel-builder build-and-upload\n```\nThe target repo is set by `repo-id` under `[general.hub]` and `version` under `[general]` in `build.toml`. Uploads go to a **`kernel`-type** Hub repository (not a model repo); the owning user\u002Forg needs kernel-creation access (\"Request Kernels Creation\" at [huggingface.co\u002Fsettings\u002Faccount](https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Faccount)).\n\n### Local build for development\nNever hand-write a `setup.py` (it leads to `torch.utils.cpp_extension`\u002Fpybind11, which cannot build under ABI3). Let kernel-builder generate the project files, then build with `setup.py build_kernel` (no `pip install`\u002Feditable install needed):\n```bash\nkernel-builder create-pyproject -f\npython setup.py build_kernel\n```\nThis builds the kernel and puts the output in `build`, which can be loaded directly with `kernels.get_local_kernel(Path(\"build\"))`. Inside `kernel-builder devshell`\u002F`testshell`, `LOCAL_KERNELS` is set automatically so `get_kernel(\"\u003Crepo-id>\")` resolves to this local build.\n\n### build.toml Configuration\n```toml\n[general]\n# Name MUST be dash-separated lowercase (my-kernel), never underscores —\n# `kernel-builder check-config` rejects underscores. The Python package\n# lives at torch-ext\u002F\u003Cname with dashes replaced by underscores>.\nname = \"ltx-kernels\"\nbackends = [\"cuda\"]\nversion = 1\nlicense = \"Apache-2.0\"   # required field\n\n[general.hub]\n# Hub repo for `kernel-builder build-and-upload`; with `version` this\n# selects the version branch (e.g. v1).\nrepo-id = \"my-username\u002Fltx-kernels\"\n\n[torch]\nsrc = [\n  \"torch-ext\u002Ftorch_binding.cpp\",\n  \"torch-ext\u002Ftorch_binding.h\"\n]\n\n[kernel.your_kernel]\nbackend = \"cuda\"\nsrc = [\"kernel_src\u002Fyour_kernel.cu\"]\ndepends = [\"torch\"]\n# Only constrain cuda-capabilities when the kernel truly requires it —\n# do not over-specify.\n```\n\nThe kernel directory **must be a git repository with files committed** (`git init && git add -A && git commit`) — Nix refuses to build non-git kernels (\"Kernel is not in a git repository\").\n\n## Library Integration\n\n### HuggingFace Kernels Hub (get_kernel)\n\n> **See [huggingface-kernels-integration.md](references\u002Fhuggingface-kernels-integration.md) for the complete guide.**\n\nLoad pre-compiled, optimized kernels directly from HuggingFace Hub without local compilation:\n\n```python\nfrom kernels import get_kernel, has_kernel\n\n# Check availability and load — Hub loads REQUIRE version= (or revision=);\n# a bare get_kernel(repo_id) raises ValueError.\nif has_kernel(\"kernels-community\u002Factivation\", version=1):\n    activation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n\n    # Use the kernel\n    x = torch.randn((4, 4), dtype=torch.float16, device=\"cuda\")\n    y = torch.empty_like(x)\n    activation.gelu_fast(y, x)\n```\n\n**Key functions:**\n- `get_kernel(repo_id, version=N)` - Download and load kernel from Hub; `version=` (major version) or `revision=` (branch\u002Ftag\u002Fcommit) is **required**\n- `has_kernel(repo_id, version=N)` - Check if compatible build exists\n- `get_local_kernel(Path(\"path\u002Fto\u002Fkernel-project\"))` - Load a local build (looks in `\u003Cpath>` and `\u003Cpath>\u002Fbuild`) — use during development\n\n**Testing local builds through the `get_kernel()` code path:** set `LOCAL_KERNELS=\"org\u002Fname=\u002Fpath\u002Fto\u002Fkernel-project\"` and call `get_kernel(\"org\u002Fname\")` unchanged — the override short-circuits the Hub entirely (no download, no version needed), so integration code can be tested verbatim against a local build.\n\n**Popular community kernels:**\n- `kernels-community\u002Factivation` - GELU, SiLU, etc.\n- `kernels-community\u002Fflash-attn` - Flash Attention 2\n- `kernels-community\u002Ftriton-layer-norm` - LayerNorm, RMSNorm\n\n### Diffusers Integration (Video\u002FImage Generation)\n\n> **See [diffusers-integration.md](references\u002Fdiffusers-integration.md) for the complete guide.**\n\n### Transformers Integration (LLMs)\n\n> **See [transformers-integration.md](references\u002Ftransformers-integration.md) for the complete guide.**\n\n**Key differences from diffusers:**\n- Transformers RMSNorm **always** has weights (no `elementwise_affine=False`)\n- Use `'RMSNorm' in class_name` to match LlamaRMSNorm, MistralRMSNorm, etc.\n- Check for `variance_epsilon` (LLaMA) or `eps` (others) for epsilon\n- No `set_processor()` pattern - use Flash Attention 2 instead\n\n**Minimal transformers pattern:**\n```python\nfrom transformers import AutoModelForCausalLM\nfrom ltx_kernels import rmsnorm\n\ndef patch_rmsnorm(model):\n    for name, module in model.named_modules():\n        if 'RMSNorm' in type(module).__name__:\n            eps = getattr(module, 'variance_epsilon', None) or getattr(module, 'eps', 1e-6)\n            def make_forward(mod, epsilon):\n                def forward(x):\n                    return rmsnorm(x, mod.weight, eps=epsilon)\n                return forward\n            module.forward = make_forward(module, eps)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\", torch_dtype=torch.bfloat16)\npatch_rmsnorm(model)\n```\n\n### Diffusers Critical Pitfalls\n\n#### 1. RMSNorm Weight May Be None\n\nLTX-Video uses `elementwise_affine=False` for some RMSNorm modules:\n```python\n# Transformer blocks: NO WEIGHT\nself.norm1 = RMSNorm(dim, elementwise_affine=False)\n\n# Attention modules: HAS WEIGHT\nself.norm_q = torch.nn.RMSNorm(..., elementwise_affine=True)\n```\n\n**Solution:** Handle both cases:\n```python\nhas_weight = hasattr(module, 'weight') and module.weight is not None\nif has_weight:\n    output = rmsnorm(x, module.weight, eps=eps)\nelse:\n    weight = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n    output = rmsnorm(x, weight, eps=eps)\n```\n\n#### 2. Diffusers RMSNorm != torch.nn.RMSNorm\n\n```python\n# WRONG - misses diffusers RMSNorm\nif isinstance(module, torch.nn.RMSNorm):\n\n# CORRECT - catches all RMSNorm variants\nif type(module).__name__ == 'RMSNorm':\n```\n\n#### 3. LTX-Video Uses GELU, Not GEGLU\n\nLTX-Video uses `activation_fn=\"gelu-approximate\"`. Don't patch GEGLU for LTX-Video.\n\n#### 4. Inject Kernels BEFORE CPU Offloading\n\n```python\npipe = LTXPipeline.from_pretrained(...)\npipe.to(\"cuda\")\ninject_optimized_kernels(pipe)  # BEFORE offloading\npipe.enable_model_cpu_offload()  # Now safe\n```\n\n### Minimal Integration Pattern\n\n```python\nfrom diffusers import LTXPipeline\nfrom ltx_kernels import rmsnorm\n\ndef patch_rmsnorm_modules(model):\n    \"\"\"Patch all RMSNorm modules to use custom kernel.\"\"\"\n    for name, module in model.named_modules():\n        if type(module).__name__ == 'RMSNorm':\n            eps = getattr(module, 'eps', 1e-6)\n            has_weight = hasattr(module, 'weight') and module.weight is not None\n\n            if has_weight:\n                def make_forward(mod, epsilon):\n                    def forward(x):\n                        return rmsnorm(x, mod.weight, eps=epsilon)\n                    return forward\n                module.forward = make_forward(module, eps)\n            else:\n                def make_forward(epsilon):\n                    def forward(x):\n                        w = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n                        return rmsnorm(x, w, eps=epsilon)\n                    return forward\n                module.forward = make_forward(eps)\n\n# Usage\npipe = LTXPipeline.from_pretrained(\"Lightricks\u002FLTX-Video\", torch_dtype=torch.bfloat16)\npipe.to(\"cuda\")\npatch_rmsnorm_modules(pipe.transformer)\npipe.enable_model_cpu_offload()\n```\n\n## Kernel-Specific Guidelines\n\n### RMSNorm\n- Input layout: `[..., hidden_size]`\n- Epsilon default: 1e-6\n- **Weight may be None** if `elementwise_affine=False`\n- **Vectorization:** Use `__nv_bfloat162` for BF16, `__half2` for FP16, `float4` for FP32\n- **Performance:** 2.67x faster than PyTorch with vectorized implementation\n- **Bandwidth:** Achieves ~38% of H100's 3.35 TB\u002Fs theoretical bandwidth\n\n### RoPE\n- 1D: `[batch, seq, heads, head_dim]` - for text\n- 3D: `[batch, t*h*w, heads, head_dim]` - for video\n- LTX-Video computes its own RoPE via `LTXVideoRotaryPosEmbed`\n\n### GEGLU vs GELU\n- **GEGLU**: Input `[batch, seq, 2*hidden]` -> Output `[batch, seq, hidden]`\n- **GELU**: Standard activation\n- **LTX-Video uses GELU, NOT GEGLU**\n\n### AdaLN\n- Formula: `norm(x) * weight * (1 + scale) + shift`\n- Used in DiT blocks for conditioning\n\n## Performance Profiling\n\n```bash\n# NVIDIA Nsight Systems\nnsys profile -o profile python your_script.py\n\n# NVIDIA Nsight Compute\nncu --set full -o metrics python your_script.py\n```\n\n## Common Issues\n\n> **See [troubleshooting.md](references\u002Ftroubleshooting.md) for all common issues and solutions.**\n\nQuick fixes:\n- **\"NoneType has no attribute contiguous\"**: RMSNorm weight is None, create ones\n- **isinstance() not matching**: Use `type(module).__name__` instead\n- **GEGLU not called**: Model uses GELU, not GEGLU\n- **Patching doesn't persist**: Inject before `enable_model_cpu_offload()`\n- **torch.compile fails with custom kernels**: See below\n\n### torch.compile Compatibility\n\nCustom CUDA kernels and `torch.compile` are **mutually exclusive** unless you register the kernel as a PyTorch custom op.\n\n**Error message:**\n```\ntorch._dynamo.exc.Unsupported: Attempted to call function marked as skipped\n```\n\n**Workaround options:**\n1. Use `--use-optimized-kernels` without `--compile` (6% speedup)\n2. Use `--compile` without custom kernels (34% speedup)\n3. Add a fake\u002Fmeta implementation for the C++-registered op (see below)\n\n**To make the op torch.compile-compatible:** ops registered via `TORCH_LIBRARY_EXPAND` in C++ are already proper custom ops — do NOT re-wrap them with `@torch.library.custom_op` in Python. Just register a fake (meta) implementation using the generated `_ops.py` helpers:\n```python\nimport torch\nfrom ._ops import ops, add_op_namespace_prefix\n\n@torch.library.register_fake(add_op_namespace_prefix(\"rmsnorm_forward\"))\ndef _(out, input, weight, eps):\n    return None  # out-variant op: no shape changes\n```\n\n## See Also\n\n### Scripts\n- [benchmark_example.py](scripts\u002Fbenchmark_example.py) - **Benchmarking script for comparing optimized vs baseline - START HERE**\n- [ltx_kernel_injection_example.py](scripts\u002Fltx_kernel_injection_example.py) - Minimal diffusers integration (~150 lines)\n- [transformers_injection_example.py](scripts\u002Ftransformers_injection_example.py) - Minimal transformers\u002FLLM integration (~120 lines)\n- [huggingface_kernels_example.py](scripts\u002Fhuggingface_kernels_example.py) - HuggingFace Kernels Hub integration\n\n### Integration Guides\n- [huggingface-kernels-integration.md](references\u002Fhuggingface-kernels-integration.md) - **HuggingFace Kernels Hub (get_kernel) - load pre-compiled kernels**\n- [diffusers-integration.md](references\u002Fdiffusers-integration.md) - Complete diffusers pipeline integration\n- [transformers-integration.md](references\u002Ftransformers-integration.md) - Complete transformers\u002FLLM integration\n\n### GPU Optimization Guides\n- [h100-optimization-guide.md](references\u002Fh100-optimization-guide.md) - H100 (Hopper, sm_90) deep dive\n- [a100-optimization-guide.md](references\u002Fa100-optimization-guide.md) - A100 (Ampere, sm_80) deep dive\n- [t4-optimization-guide.md](references\u002Ft4-optimization-guide.md) - T4 (Turing, sm_75) deep dive\n\n### Reference\n- [troubleshooting.md](references\u002Ftroubleshooting.md) - Common issues and solutions\n- [kernel-templates.md](references\u002Fkernel-templates.md) - Complete kernel templates\n- [examples\u002Fkernels\u002Frelu\u002F](..\u002F..\u002F..\u002Fexamples\u002Fkernels\u002Frelu\u002F) - Canonical working kernel example (bindings, layers, tests)\n\n### External Resources\n- [HuggingFace Kernels Documentation](https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fkernels\u002Fen\u002Findex)\n- [HuggingFace Kernels GitHub](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels)\n- [Community Kernels on Hub](https:\u002F\u002Fhuggingface.co\u002Fkernels-community)\n",{"data":39,"body":44},{"name":4,"description":6,"disable-model-invocation":40,"user-invocable":41,"allowed-tools":42,"argument-hint":43},false,true,"Read, Grep, Glob, Bash","kernel type: attention, rmsnorm, rope, adaln, geglu, benchmark, transformers, diffusers, huggingface-kernels, get_kernel",{"type":45,"children":46},"root",[47,56,77,84,122,129,678,684,708,722,771,783,874,887,950,956,1097,1103,1109,1117,1252,1260,1279,1285,1293,1312,1318,1326,1395,1403,1422,1428,1447,1453,1520,1613,1619,1624,1679,1685,1707,1766,1772,1777,1931,1937,2162,2168,2176,2305,2327,2335,2378,2384,2396,2572,2582,2592,2598,2608,2614,2620,2755,2761,2899,2923,2929,2935,2950,3037,3052,3091,3106,3137,3143,3213,3219,3254,3260,3265,3288,3293,3332,3338,3343,3379,3385,3391,3404,3433,3527,3533,3583,3589,3608,3670,3676,3711,3752,3804,3810,4040,4059,4065,4071,4090,4095,4188,4196,4268,4302,4310,4346,4352,4369,4375,4392,4400,4468,4476,4601,4607,4614,4626,4672,4682,4737,4743,4789,4795,4807,4813,4852,4858,5092,5098,5104,5189,5195,5235,5241,5286,5292,5311,5317,5414,5420,5438,5443,5509,5515,5533,5541,5550,5558,5595,5626,5679,5685,5691,5743,5749,5783,5789,5819,5824,5858,5864,5896],{"type":48,"tag":49,"props":50,"children":52},"element","h1",{"id":51},"cuda-kernels-for-diffusers-transformers",[53],{"type":54,"value":55},"text","CUDA Kernels for Diffusers & Transformers",{"type":48,"tag":57,"props":58,"children":59},"p",{},[60,62,68,70,75],{"type":54,"value":61},"This skill provides patterns and guidance for developing optimized CUDA kernels targeting NVIDIA GPUs (H100, A100, T4) for use with HuggingFace ",{"type":48,"tag":63,"props":64,"children":65},"strong",{},[66],{"type":54,"value":67},"diffusers",{"type":54,"value":69}," and ",{"type":48,"tag":63,"props":71,"children":72},{},[73],{"type":54,"value":74},"transformers",{"type":54,"value":76}," libraries.",{"type":48,"tag":78,"props":79,"children":81},"h2",{"id":80},"hard-constraints-read-before-writing-any-code",[82],{"type":54,"value":83},"Hard Constraints — Read Before Writing Any Code",{"type":48,"tag":57,"props":85,"children":86},{},[87,89,97,99,106,108,113,115,120],{"type":54,"value":88},"Kernels MUST build with ",{"type":48,"tag":90,"props":91,"children":94},"a",{"href":28,"rel":92},[93],"nofollow",[95],{"type":54,"value":96},"kernel-builder",{"type":54,"value":98}," and meet the ",{"type":48,"tag":90,"props":100,"children":103},{"href":101,"rel":102},"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fkernels\u002Fkernel-requirements",[93],[104],{"type":54,"value":105},"Kernel Hub requirements",{"type":54,"value":107},". kernel-builder compiles against the ",{"type":48,"tag":63,"props":109,"children":110},{},[111],{"type":54,"value":112},"Python limited API (ABI3)",{"type":54,"value":114}," so a single binary works for Python 3.9+ across versions. Several patterns that are standard in generic PyTorch-extension tutorials are therefore ",{"type":48,"tag":63,"props":116,"children":117},{},[118],{"type":54,"value":119},"hard build failures",{"type":54,"value":121}," here. Do not use them, even if PyTorch documentation or your training data suggests them.",{"type":48,"tag":123,"props":124,"children":126},"h3",{"id":125},"disallowed-patterns-never-generate-these",[127],{"type":54,"value":128},"Disallowed patterns — never generate these",{"type":48,"tag":130,"props":131,"children":132},"table",{},[133,157],{"type":48,"tag":134,"props":135,"children":136},"thead",{},[137],{"type":48,"tag":138,"props":139,"children":140},"tr",{},[141,147,152],{"type":48,"tag":142,"props":143,"children":144},"th",{},[145],{"type":54,"value":146},"❌ Never use",{"type":48,"tag":142,"props":148,"children":149},{},[150],{"type":54,"value":151},"Why it fails",{"type":48,"tag":142,"props":153,"children":154},{},[155],{"type":54,"value":156},"✅ Use instead",{"type":48,"tag":158,"props":159,"children":160},"tbody",{},[161,257,356,415,453,498,552,596,637],{"type":48,"tag":138,"props":162,"children":163},{},[164,209,214],{"type":48,"tag":165,"props":166,"children":167},"td",{},[168,170,177,179,185,186,192,193,199,201,207],{"type":54,"value":169},"pybind11 in any form: ",{"type":48,"tag":171,"props":172,"children":174},"code",{"className":173},[],[175],{"type":54,"value":176},"#include \u003Ctorch\u002Fextension.h>",{"type":54,"value":178},", ",{"type":48,"tag":171,"props":180,"children":182},{"className":181},[],[183],{"type":54,"value":184},"#include \u003Cpybind11\u002F...>",{"type":54,"value":178},{"type":48,"tag":171,"props":187,"children":189},{"className":188},[],[190],{"type":54,"value":191},"PYBIND11_MODULE(...)",{"type":54,"value":178},{"type":48,"tag":171,"props":194,"children":196},{"className":195},[],[197],{"type":54,"value":198},"py::arg",{"type":54,"value":200},", any ",{"type":48,"tag":171,"props":202,"children":204},{"className":203},[],[205],{"type":54,"value":206},"py::",{"type":54,"value":208}," symbol",{"type":48,"tag":165,"props":210,"children":211},{},[212],{"type":54,"value":213},"pybind11 is incompatible with the limited API (ABI3); the build does not compile",{"type":48,"tag":165,"props":215,"children":216},{},[217,223,225,231,233,239,241,247,249,255],{"type":48,"tag":171,"props":218,"children":220},{"className":219},[],[221],{"type":54,"value":222},"TORCH_LIBRARY_EXPAND",{"type":54,"value":224}," in ",{"type":48,"tag":171,"props":226,"children":228},{"className":227},[],[229],{"type":54,"value":230},"torch-ext\u002Ftorch_binding.cpp",{"type":54,"value":232}," (see below). Note: ",{"type":48,"tag":171,"props":234,"children":236},{"className":235},[],[237],{"type":54,"value":238},"torch\u002Fextension.h",{"type":54,"value":240}," transitively includes pybind11 — include ",{"type":48,"tag":171,"props":242,"children":244},{"className":243},[],[245],{"type":54,"value":246},"torch\u002Ftorch.h",{"type":54,"value":248}," + ",{"type":48,"tag":171,"props":250,"children":252},{"className":251},[],[253],{"type":54,"value":254},"torch\u002Flibrary.h",{"type":54,"value":256}," instead",{"type":48,"tag":138,"props":258,"children":259},{},[260,318,331],{"type":48,"tag":165,"props":261,"children":262},{},[263,265,271,273,279,281,287,289,295,296,302,303,309,310,316],{"type":54,"value":264},"Hand-written ",{"type":48,"tag":171,"props":266,"children":268},{"className":267},[],[269],{"type":54,"value":270},"setup.py",{"type":54,"value":272}," \u002F ",{"type":48,"tag":171,"props":274,"children":276},{"className":275},[],[277],{"type":54,"value":278},"pyproject.toml",{"type":54,"value":280}," using ",{"type":48,"tag":171,"props":282,"children":284},{"className":283},[],[285],{"type":54,"value":286},"torch.utils.cpp_extension",{"type":54,"value":288}," (",{"type":48,"tag":171,"props":290,"children":292},{"className":291},[],[293],{"type":54,"value":294},"CUDAExtension",{"type":54,"value":178},{"type":48,"tag":171,"props":297,"children":299},{"className":298},[],[300],{"type":54,"value":301},"BuildExtension",{"type":54,"value":178},{"type":48,"tag":171,"props":304,"children":306},{"className":305},[],[307],{"type":54,"value":308},"cpp_extension.load",{"type":54,"value":178},{"type":48,"tag":171,"props":311,"children":313},{"className":312},[],[314],{"type":54,"value":315},"load_inline",{"type":54,"value":317},")",{"type":48,"tag":165,"props":319,"children":320},{},[321,323,329],{"type":54,"value":322},"setuptools extensions are not ABI3 and bypass ",{"type":48,"tag":171,"props":324,"children":326},{"className":325},[],[327],{"type":54,"value":328},"build.toml",{"type":54,"value":330},"; kernel-builder owns the build",{"type":48,"tag":165,"props":332,"children":333},{},[334,339,340,346,348,354],{"type":48,"tag":171,"props":335,"children":337},{"className":336},[],[338],{"type":54,"value":328},{"type":54,"value":248},{"type":48,"tag":171,"props":341,"children":343},{"className":342},[],[344],{"type":54,"value":345},"nix run .#build-and-copy -L",{"type":54,"value":347},". For an editable dev install, generate the project files with ",{"type":48,"tag":171,"props":349,"children":351},{"className":350},[],[352],{"type":54,"value":353},"kernel-builder create-pyproject -f",{"type":54,"value":355}," — never write them by hand",{"type":48,"tag":138,"props":357,"children":358},{},[359,385,398],{"type":48,"tag":165,"props":360,"children":361},{},[362,368,369,375,377,383],{"type":48,"tag":171,"props":363,"children":365},{"className":364},[],[366],{"type":54,"value":367},"TORCH_LIBRARY(my_kernel, m)",{"type":54,"value":178},{"type":48,"tag":171,"props":370,"children":372},{"className":371},[],[373],{"type":54,"value":374},"TORCH_LIBRARY_FRAGMENT(...)",{"type":54,"value":376},", or ",{"type":48,"tag":171,"props":378,"children":380},{"className":379},[],[381],{"type":54,"value":382},"TORCH_LIBRARY_IMPL(...)",{"type":54,"value":384}," with a hardcoded namespace",{"type":48,"tag":165,"props":386,"children":387},{},[388,390,396],{"type":54,"value":389},"kernel-builder suffixes the op namespace with a per-build hash (e.g. ",{"type":48,"tag":171,"props":391,"children":393},{"className":392},[],[394],{"type":54,"value":395},"_my_kernel_a1b2c3d",{"type":54,"value":397},"); a hardcoded name never resolves",{"type":48,"tag":165,"props":399,"children":400},{},[401,407,409],{"type":48,"tag":171,"props":402,"children":404},{"className":403},[],[405],{"type":54,"value":406},"TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops)",{"type":54,"value":408}," from the generated ",{"type":48,"tag":171,"props":410,"children":412},{"className":411},[],[413],{"type":54,"value":414},"registration.h",{"type":48,"tag":138,"props":416,"children":417},{},[418,431,436],{"type":48,"tag":165,"props":419,"children":420},{},[421,423,429],{"type":54,"value":422},"Hardcoded ",{"type":48,"tag":171,"props":424,"children":426},{"className":425},[],[427],{"type":54,"value":428},"torch.ops.my_kernel.fn(...)",{"type":54,"value":430}," calls in Python",{"type":48,"tag":165,"props":432,"children":433},{},[434],{"type":54,"value":435},"Same namespace mangling — the op namespace name is only known at build time",{"type":48,"tag":165,"props":437,"children":438},{},[439,445,447],{"type":48,"tag":171,"props":440,"children":442},{"className":441},[],[443],{"type":54,"value":444},"from ._ops import ops",{"type":54,"value":446}," then ",{"type":48,"tag":171,"props":448,"children":450},{"className":449},[],[451],{"type":54,"value":452},"ops.fn(...)",{"type":48,"tag":138,"props":454,"children":455},{},[456,468,481],{"type":48,"tag":165,"props":457,"children":458},{},[459,460,466],{"type":54,"value":264},{"type":48,"tag":171,"props":461,"children":463},{"className":462},[],[464],{"type":54,"value":465},"PyMODINIT_FUNC PyInit__...",{"type":54,"value":467}," or any manual CPython module init",{"type":48,"tag":165,"props":469,"children":470},{},[471,473,479],{"type":54,"value":472},"Generated by ",{"type":48,"tag":171,"props":474,"children":476},{"className":475},[],[477],{"type":54,"value":478},"REGISTER_EXTENSION",{"type":54,"value":480},"; duplicating it breaks module loading",{"type":48,"tag":165,"props":482,"children":483},{},[484,490,492],{"type":48,"tag":171,"props":485,"children":487},{"className":486},[],[488],{"type":54,"value":489},"REGISTER_EXTENSION(TORCH_EXTENSION_NAME)",{"type":54,"value":491}," exactly once, in ",{"type":48,"tag":171,"props":493,"children":495},{"className":494},[],[496],{"type":54,"value":497},"torch_binding.cpp",{"type":48,"tag":138,"props":499,"children":500},{},[501,522,527],{"type":48,"tag":165,"props":502,"children":503},{},[504,506,512,514,520],{"type":54,"value":505},"Non-limited CPython API calls (",{"type":48,"tag":171,"props":507,"children":509},{"className":508},[],[510],{"type":54,"value":511},"PyArg_ParseTuple",{"type":54,"value":513},", direct ",{"type":48,"tag":171,"props":515,"children":517},{"className":516},[],[518],{"type":54,"value":519},"PyObject*",{"type":54,"value":521}," manipulation)",{"type":48,"tag":165,"props":523,"children":524},{},[525],{"type":54,"value":526},"Violates ABI3",{"type":48,"tag":165,"props":528,"children":529},{},[530,532,538,539,545,546],{"type":54,"value":531},"Stay within the torch C++ API: ",{"type":48,"tag":171,"props":533,"children":535},{"className":534},[],[536],{"type":54,"value":537},"torch::Tensor",{"type":54,"value":178},{"type":48,"tag":171,"props":540,"children":542},{"className":541},[],[543],{"type":54,"value":544},"TORCH_CHECK",{"type":54,"value":178},{"type":48,"tag":171,"props":547,"children":549},{"className":548},[],[550],{"type":54,"value":551},"at::cuda::*",{"type":48,"tag":138,"props":553,"children":554},{},[555,574,579],{"type":48,"tag":165,"props":556,"children":557},{},[558,560,566,567,573],{"type":54,"value":559},"Absolute imports of your own package inside ",{"type":48,"tag":171,"props":561,"children":563},{"className":562},[],[564],{"type":54,"value":565},"torch-ext\u002F",{"type":54,"value":288},{"type":48,"tag":171,"props":568,"children":570},{"className":569},[],[571],{"type":54,"value":572},"from my_kernel.utils import x",{"type":54,"value":317},{"type":48,"tag":165,"props":575,"children":576},{},[577],{"type":54,"value":578},"The package directory is renamed when loaded from the Hub; absolute imports break",{"type":48,"tag":165,"props":580,"children":581},{},[582,584,590,591],{"type":54,"value":583},"Relative imports only: ",{"type":48,"tag":171,"props":585,"children":587},{"className":586},[],[588],{"type":54,"value":589},"from .utils import x",{"type":54,"value":178},{"type":48,"tag":171,"props":592,"children":594},{"className":593},[],[595],{"type":54,"value":444},{"type":48,"tag":138,"props":597,"children":598},{},[599,620,625],{"type":48,"tag":165,"props":600,"children":601},{},[602,604,610,612,618],{"type":54,"value":603},"Runtime Python deps beyond ",{"type":48,"tag":171,"props":605,"children":607},{"className":606},[],[608],{"type":54,"value":609},"torch",{"type":54,"value":611}," (and ",{"type":48,"tag":171,"props":613,"children":615},{"className":614},[],[616],{"type":54,"value":617},"einops",{"type":54,"value":619}," if truly needed)",{"type":48,"tag":165,"props":621,"children":622},{},[623],{"type":54,"value":624},"Hub compliance restricts kernel dependencies; imports of numpy, triton, packaging, etc. are rejected",{"type":48,"tag":165,"props":626,"children":627},{},[628,630,635],{"type":54,"value":629},"Standard library + ",{"type":48,"tag":171,"props":631,"children":633},{"className":632},[],[634],{"type":54,"value":609},{"type":54,"value":636}," only",{"type":48,"tag":138,"props":638,"children":639},{},[640,653,658],{"type":48,"tag":165,"props":641,"children":642},{},[643,645,651],{"type":54,"value":644},"Python-side ",{"type":48,"tag":171,"props":646,"children":648},{"className":647},[],[649],{"type":54,"value":650},"@torch.library.custom_op",{"type":54,"value":652}," as the primary binding",{"type":48,"tag":165,"props":654,"children":655},{},[656],{"type":54,"value":657},"The op must be registered in C++ so it ships in the compiled extension",{"type":48,"tag":165,"props":659,"children":660},{},[661,663,668,670,676],{"type":54,"value":662},"C++ registration via ",{"type":48,"tag":171,"props":664,"children":666},{"className":665},[],[667],{"type":54,"value":222},{"type":54,"value":669},"; Python-side ",{"type":48,"tag":171,"props":671,"children":673},{"className":672},[],[674],{"type":54,"value":675},"torch.library.register_fake",{"type":54,"value":677}," is only for adding a fake\u002Fmeta impl (see torch.compile section)",{"type":48,"tag":123,"props":679,"children":681},{"id":680},"the-only-supported-binding-pattern",[682],{"type":54,"value":683},"The only supported binding pattern",{"type":48,"tag":57,"props":685,"children":686},{},[687,692,693,699,701,706],{"type":48,"tag":171,"props":688,"children":690},{"className":689},[],[691],{"type":54,"value":414},{"type":54,"value":69},{"type":48,"tag":171,"props":694,"children":696},{"className":695},[],[697],{"type":54,"value":698},"_ops.py",{"type":54,"value":700}," are ",{"type":48,"tag":63,"props":702,"children":703},{},[704],{"type":54,"value":705},"generated by kernel-builder",{"type":54,"value":707}," — reference them, never write them yourself.",{"type":48,"tag":57,"props":709,"children":710},{},[711],{"type":48,"tag":63,"props":712,"children":713},{},[714,720],{"type":48,"tag":171,"props":715,"children":717},{"className":716},[],[718],{"type":54,"value":719},"torch-ext\u002Ftorch_binding.h",{"type":54,"value":721},":",{"type":48,"tag":723,"props":724,"children":729},"pre",{"className":725,"code":726,"language":727,"meta":728,"style":728},"language-cpp shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","#pragma once\n#include \u003Ctorch\u002Ftorch.h>\n\nvoid my_kernel_forward(torch::Tensor &out, torch::Tensor const &input);\n","cpp","",[730],{"type":48,"tag":171,"props":731,"children":732},{"__ignoreMap":728},[733,744,753,762],{"type":48,"tag":734,"props":735,"children":738},"span",{"class":736,"line":737},"line",1,[739],{"type":48,"tag":734,"props":740,"children":741},{},[742],{"type":54,"value":743},"#pragma once\n",{"type":48,"tag":734,"props":745,"children":747},{"class":736,"line":746},2,[748],{"type":48,"tag":734,"props":749,"children":750},{},[751],{"type":54,"value":752},"#include \u003Ctorch\u002Ftorch.h>\n",{"type":48,"tag":734,"props":754,"children":756},{"class":736,"line":755},3,[757],{"type":48,"tag":734,"props":758,"children":759},{"emptyLinePlaceholder":41},[760],{"type":54,"value":761},"\n",{"type":48,"tag":734,"props":763,"children":765},{"class":736,"line":764},4,[766],{"type":48,"tag":734,"props":767,"children":768},{},[769],{"type":54,"value":770},"void my_kernel_forward(torch::Tensor &out, torch::Tensor const &input);\n",{"type":48,"tag":57,"props":772,"children":773},{},[774],{"type":48,"tag":63,"props":775,"children":776},{},[777,782],{"type":48,"tag":171,"props":778,"children":780},{"className":779},[],[781],{"type":54,"value":230},{"type":54,"value":721},{"type":48,"tag":723,"props":784,"children":786},{"className":725,"code":785,"language":727,"meta":728,"style":728},"#include \u003Ctorch\u002Flibrary.h>\n#include \"registration.h\"\n#include \"torch_binding.h\"\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n  ops.def(\"my_kernel_forward(Tensor! out, Tensor input) -> ()\");\n  ops.impl(\"my_kernel_forward\", torch::kCUDA, &my_kernel_forward);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n",[787],{"type":48,"tag":171,"props":788,"children":789},{"__ignoreMap":728},[790,798,806,814,821,830,839,848,857,865],{"type":48,"tag":734,"props":791,"children":792},{"class":736,"line":737},[793],{"type":48,"tag":734,"props":794,"children":795},{},[796],{"type":54,"value":797},"#include \u003Ctorch\u002Flibrary.h>\n",{"type":48,"tag":734,"props":799,"children":800},{"class":736,"line":746},[801],{"type":48,"tag":734,"props":802,"children":803},{},[804],{"type":54,"value":805},"#include \"registration.h\"\n",{"type":48,"tag":734,"props":807,"children":808},{"class":736,"line":755},[809],{"type":48,"tag":734,"props":810,"children":811},{},[812],{"type":54,"value":813},"#include \"torch_binding.h\"\n",{"type":48,"tag":734,"props":815,"children":816},{"class":736,"line":764},[817],{"type":48,"tag":734,"props":818,"children":819},{"emptyLinePlaceholder":41},[820],{"type":54,"value":761},{"type":48,"tag":734,"props":822,"children":824},{"class":736,"line":823},5,[825],{"type":48,"tag":734,"props":826,"children":827},{},[828],{"type":54,"value":829},"TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n",{"type":48,"tag":734,"props":831,"children":833},{"class":736,"line":832},6,[834],{"type":48,"tag":734,"props":835,"children":836},{},[837],{"type":54,"value":838},"  ops.def(\"my_kernel_forward(Tensor! out, Tensor input) -> ()\");\n",{"type":48,"tag":734,"props":840,"children":842},{"class":736,"line":841},7,[843],{"type":48,"tag":734,"props":844,"children":845},{},[846],{"type":54,"value":847},"  ops.impl(\"my_kernel_forward\", torch::kCUDA, &my_kernel_forward);\n",{"type":48,"tag":734,"props":849,"children":851},{"class":736,"line":850},8,[852],{"type":48,"tag":734,"props":853,"children":854},{},[855],{"type":54,"value":856},"}\n",{"type":48,"tag":734,"props":858,"children":860},{"class":736,"line":859},9,[861],{"type":48,"tag":734,"props":862,"children":863},{"emptyLinePlaceholder":41},[864],{"type":54,"value":761},{"type":48,"tag":734,"props":866,"children":868},{"class":736,"line":867},10,[869],{"type":48,"tag":734,"props":870,"children":871},{},[872],{"type":54,"value":873},"REGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n",{"type":48,"tag":57,"props":875,"children":876},{},[877],{"type":48,"tag":63,"props":878,"children":879},{},[880,886],{"type":48,"tag":171,"props":881,"children":883},{"className":882},[],[884],{"type":54,"value":885},"torch-ext\u002Fmy_kernel\u002F__init__.py",{"type":54,"value":721},{"type":48,"tag":723,"props":888,"children":891},{"className":889,"code":890,"language":23,"meta":728,"style":728},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import torch\nfrom ._ops import ops\n\ndef my_kernel(x: torch.Tensor) -> torch.Tensor:\n    out = torch.empty_like(x)\n    ops.my_kernel_forward(out, x)\n    return out\n",[892],{"type":48,"tag":171,"props":893,"children":894},{"__ignoreMap":728},[895,903,911,918,926,934,942],{"type":48,"tag":734,"props":896,"children":897},{"class":736,"line":737},[898],{"type":48,"tag":734,"props":899,"children":900},{},[901],{"type":54,"value":902},"import torch\n",{"type":48,"tag":734,"props":904,"children":905},{"class":736,"line":746},[906],{"type":48,"tag":734,"props":907,"children":908},{},[909],{"type":54,"value":910},"from ._ops import ops\n",{"type":48,"tag":734,"props":912,"children":913},{"class":736,"line":755},[914],{"type":48,"tag":734,"props":915,"children":916},{"emptyLinePlaceholder":41},[917],{"type":54,"value":761},{"type":48,"tag":734,"props":919,"children":920},{"class":736,"line":764},[921],{"type":48,"tag":734,"props":922,"children":923},{},[924],{"type":54,"value":925},"def my_kernel(x: torch.Tensor) -> torch.Tensor:\n",{"type":48,"tag":734,"props":927,"children":928},{"class":736,"line":823},[929],{"type":48,"tag":734,"props":930,"children":931},{},[932],{"type":54,"value":933},"    out = torch.empty_like(x)\n",{"type":48,"tag":734,"props":935,"children":936},{"class":736,"line":832},[937],{"type":48,"tag":734,"props":938,"children":939},{},[940],{"type":54,"value":941},"    ops.my_kernel_forward(out, x)\n",{"type":48,"tag":734,"props":943,"children":944},{"class":736,"line":841},[945],{"type":48,"tag":734,"props":946,"children":947},{},[948],{"type":54,"value":949},"    return out\n",{"type":48,"tag":123,"props":951,"children":953},{"id":952},"pre-flight-checklist-before-declaring-a-kernel-done",[954],{"type":54,"value":955},"Pre-flight checklist before declaring a kernel done",{"type":48,"tag":957,"props":958,"children":959},"ol",{},[960,972,990,1010,1068,1073,1084],{"type":48,"tag":961,"props":962,"children":963},"li",{},[964,970],{"type":48,"tag":171,"props":965,"children":967},{"className":966},[],[968],{"type":54,"value":969},"grep -rn \"pybind11\\|PYBIND11\\|torch\u002Fextension.h\\|py::\" torch-ext\u002F",{"type":54,"value":971}," returns nothing.",{"type":48,"tag":961,"props":973,"children":974},{},[975,981,983,988],{"type":48,"tag":171,"props":976,"children":978},{"className":977},[],[979],{"type":54,"value":980},"grep -rn \"TORCH_LIBRARY(\\|TORCH_LIBRARY_FRAGMENT\\|PyInit\" torch-ext\u002F",{"type":54,"value":982}," returns nothing (only ",{"type":48,"tag":171,"props":984,"children":986},{"className":985},[],[987],{"type":54,"value":222},{"type":54,"value":989}," is allowed).",{"type":48,"tag":961,"props":991,"children":992},{},[993,995,1000,1002,1008],{"type":54,"value":994},"No ",{"type":48,"tag":171,"props":996,"children":998},{"className":997},[],[999],{"type":54,"value":270},{"type":54,"value":1001}," exists unless generated by ",{"type":48,"tag":171,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":54,"value":1007},"kernel-builder create-pyproject",{"type":54,"value":1009},".",{"type":48,"tag":961,"props":1011,"children":1012},{},[1013,1019,1021,1027,1029,1034,1036,1042,1044,1050,1052,1058,1060,1066],{"type":48,"tag":171,"props":1014,"children":1016},{"className":1015},[],[1017],{"type":54,"value":1018},"kernel-builder check-config",{"type":54,"value":1020}," passes — ",{"type":48,"tag":171,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":54,"value":1026},"[general]",{"type":54,"value":1028}," needs a ",{"type":48,"tag":63,"props":1030,"children":1031},{},[1032],{"type":54,"value":1033},"dash-separated",{"type":54,"value":1035}," ",{"type":48,"tag":171,"props":1037,"children":1039},{"className":1038},[],[1040],{"type":54,"value":1041},"name",{"type":54,"value":1043}," (never underscores) and a ",{"type":48,"tag":171,"props":1045,"children":1047},{"className":1046},[],[1048],{"type":54,"value":1049},"license",{"type":54,"value":1051},", plus ",{"type":48,"tag":171,"props":1053,"children":1055},{"className":1054},[],[1056],{"type":54,"value":1057},"[torch]",{"type":54,"value":1059}," (binding sources) and ",{"type":48,"tag":171,"props":1061,"children":1063},{"className":1062},[],[1064],{"type":54,"value":1065},"[kernel.\u003Cname>]",{"type":54,"value":1067}," sections.",{"type":48,"tag":961,"props":1069,"children":1070},{},[1071],{"type":54,"value":1072},"The kernel directory is a git repository with all files committed (Nix refuses non-git builds).",{"type":48,"tag":961,"props":1074,"children":1075},{},[1076,1078,1083],{"type":54,"value":1077},"The build succeeds: ",{"type":48,"tag":171,"props":1079,"children":1081},{"className":1080},[],[1082],{"type":54,"value":345},{"type":54,"value":1009},{"type":48,"tag":961,"props":1085,"children":1086},{},[1087,1089,1095],{"type":54,"value":1088},"ABI compliance passes: ",{"type":48,"tag":171,"props":1090,"children":1092},{"className":1091},[],[1093],{"type":54,"value":1094},"kernel-builder check-abi",{"type":54,"value":1096}," (after building).",{"type":48,"tag":78,"props":1098,"children":1100},{"id":1099},"quick-start",[1101],{"type":54,"value":1102},"Quick Start",{"type":48,"tag":123,"props":1104,"children":1106},{"id":1105},"diffusers-videoimage-generation",[1107],{"type":54,"value":1108},"Diffusers (Video\u002FImage Generation)",{"type":48,"tag":57,"props":1110,"children":1111},{},[1112],{"type":48,"tag":63,"props":1113,"children":1114},{},[1115],{"type":54,"value":1116},"For benchmarking kernel performance:",{"type":48,"tag":723,"props":1118,"children":1122},{"className":1119,"code":1120,"language":1121,"meta":728,"style":728},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Benchmark with optimized kernels (6% end-to-end speedup)\npython generate_video.py --use-optimized-kernels\n\n# Benchmark baseline with torch.compile (34% speedup)\npython generate_video.py --no-optimized-kernels --compile\n\n# Compare configurations (note: --compile and --use-optimized-kernels are mutually exclusive)\npython generate_video.py --use-optimized-kernels && \\\npython generate_video.py --no-optimized-kernels --compile\n","bash",[1123],{"type":48,"tag":171,"props":1124,"children":1125},{"__ignoreMap":728},[1126,1135,1154,1161,1169,1190,1197,1205,1233],{"type":48,"tag":734,"props":1127,"children":1128},{"class":736,"line":737},[1129],{"type":48,"tag":734,"props":1130,"children":1132},{"style":1131},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1133],{"type":54,"value":1134},"# Benchmark with optimized kernels (6% end-to-end speedup)\n",{"type":48,"tag":734,"props":1136,"children":1137},{"class":736,"line":746},[1138,1143,1149],{"type":48,"tag":734,"props":1139,"children":1141},{"style":1140},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1142],{"type":54,"value":23},{"type":48,"tag":734,"props":1144,"children":1146},{"style":1145},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1147],{"type":54,"value":1148}," generate_video.py",{"type":48,"tag":734,"props":1150,"children":1151},{"style":1145},[1152],{"type":54,"value":1153}," --use-optimized-kernels\n",{"type":48,"tag":734,"props":1155,"children":1156},{"class":736,"line":755},[1157],{"type":48,"tag":734,"props":1158,"children":1159},{"emptyLinePlaceholder":41},[1160],{"type":54,"value":761},{"type":48,"tag":734,"props":1162,"children":1163},{"class":736,"line":764},[1164],{"type":48,"tag":734,"props":1165,"children":1166},{"style":1131},[1167],{"type":54,"value":1168},"# Benchmark baseline with torch.compile (34% speedup)\n",{"type":48,"tag":734,"props":1170,"children":1171},{"class":736,"line":823},[1172,1176,1180,1185],{"type":48,"tag":734,"props":1173,"children":1174},{"style":1140},[1175],{"type":54,"value":23},{"type":48,"tag":734,"props":1177,"children":1178},{"style":1145},[1179],{"type":54,"value":1148},{"type":48,"tag":734,"props":1181,"children":1182},{"style":1145},[1183],{"type":54,"value":1184}," --no-optimized-kernels",{"type":48,"tag":734,"props":1186,"children":1187},{"style":1145},[1188],{"type":54,"value":1189}," --compile\n",{"type":48,"tag":734,"props":1191,"children":1192},{"class":736,"line":832},[1193],{"type":48,"tag":734,"props":1194,"children":1195},{"emptyLinePlaceholder":41},[1196],{"type":54,"value":761},{"type":48,"tag":734,"props":1198,"children":1199},{"class":736,"line":841},[1200],{"type":48,"tag":734,"props":1201,"children":1202},{"style":1131},[1203],{"type":54,"value":1204},"# Compare configurations (note: --compile and --use-optimized-kernels are mutually exclusive)\n",{"type":48,"tag":734,"props":1206,"children":1207},{"class":736,"line":850},[1208,1212,1216,1221,1227],{"type":48,"tag":734,"props":1209,"children":1210},{"style":1140},[1211],{"type":54,"value":23},{"type":48,"tag":734,"props":1213,"children":1214},{"style":1145},[1215],{"type":54,"value":1148},{"type":48,"tag":734,"props":1217,"children":1218},{"style":1145},[1219],{"type":54,"value":1220}," --use-optimized-kernels",{"type":48,"tag":734,"props":1222,"children":1224},{"style":1223},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1225],{"type":54,"value":1226}," &&",{"type":48,"tag":734,"props":1228,"children":1230},{"style":1229},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1231],{"type":54,"value":1232}," \\\n",{"type":48,"tag":734,"props":1234,"children":1235},{"class":736,"line":859},[1236,1240,1244,1248],{"type":48,"tag":734,"props":1237,"children":1238},{"style":1140},[1239],{"type":54,"value":23},{"type":48,"tag":734,"props":1241,"children":1242},{"style":1145},[1243],{"type":54,"value":1148},{"type":48,"tag":734,"props":1245,"children":1246},{"style":1145},[1247],{"type":54,"value":1184},{"type":48,"tag":734,"props":1249,"children":1250},{"style":1145},[1251],{"type":54,"value":1189},{"type":48,"tag":57,"props":1253,"children":1254},{},[1255],{"type":48,"tag":63,"props":1256,"children":1257},{},[1258],{"type":54,"value":1259},"For a minimal diffusers integration example (~150 lines):",{"type":48,"tag":723,"props":1261,"children":1263},{"className":1119,"code":1262,"language":1121,"meta":728,"style":728},"python scripts\u002Fltx_kernel_injection_example.py\n",[1264],{"type":48,"tag":171,"props":1265,"children":1266},{"__ignoreMap":728},[1267],{"type":48,"tag":734,"props":1268,"children":1269},{"class":736,"line":737},[1270,1274],{"type":48,"tag":734,"props":1271,"children":1272},{"style":1140},[1273],{"type":54,"value":23},{"type":48,"tag":734,"props":1275,"children":1276},{"style":1145},[1277],{"type":54,"value":1278}," scripts\u002Fltx_kernel_injection_example.py\n",{"type":48,"tag":123,"props":1280,"children":1282},{"id":1281},"transformers-llms",[1283],{"type":54,"value":1284},"Transformers (LLMs)",{"type":48,"tag":57,"props":1286,"children":1287},{},[1288],{"type":48,"tag":63,"props":1289,"children":1290},{},[1291],{"type":54,"value":1292},"For a minimal transformers integration example (~120 lines):",{"type":48,"tag":723,"props":1294,"children":1296},{"className":1119,"code":1295,"language":1121,"meta":728,"style":728},"python scripts\u002Ftransformers_injection_example.py\n",[1297],{"type":48,"tag":171,"props":1298,"children":1299},{"__ignoreMap":728},[1300],{"type":48,"tag":734,"props":1301,"children":1302},{"class":736,"line":737},[1303,1307],{"type":48,"tag":734,"props":1304,"children":1305},{"style":1140},[1306],{"type":54,"value":23},{"type":48,"tag":734,"props":1308,"children":1309},{"style":1145},[1310],{"type":54,"value":1311}," scripts\u002Ftransformers_injection_example.py\n",{"type":48,"tag":123,"props":1313,"children":1315},{"id":1314},"huggingface-kernels-hub",[1316],{"type":54,"value":1317},"HuggingFace Kernels Hub",{"type":48,"tag":57,"props":1319,"children":1320},{},[1321],{"type":48,"tag":63,"props":1322,"children":1323},{},[1324],{"type":54,"value":1325},"Load pre-compiled kernels from HuggingFace Hub (no local compilation):",{"type":48,"tag":723,"props":1327,"children":1329},{"className":889,"code":1328,"language":23,"meta":728,"style":728},"from kernels import get_kernel\n\n# Load optimized activation kernels\nactivation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n\n# Use the kernel\ny = torch.empty_like(x)\nactivation.gelu_fast(y, x)\n",[1330],{"type":48,"tag":171,"props":1331,"children":1332},{"__ignoreMap":728},[1333,1341,1348,1356,1364,1371,1379,1387],{"type":48,"tag":734,"props":1334,"children":1335},{"class":736,"line":737},[1336],{"type":48,"tag":734,"props":1337,"children":1338},{},[1339],{"type":54,"value":1340},"from kernels import get_kernel\n",{"type":48,"tag":734,"props":1342,"children":1343},{"class":736,"line":746},[1344],{"type":48,"tag":734,"props":1345,"children":1346},{"emptyLinePlaceholder":41},[1347],{"type":54,"value":761},{"type":48,"tag":734,"props":1349,"children":1350},{"class":736,"line":755},[1351],{"type":48,"tag":734,"props":1352,"children":1353},{},[1354],{"type":54,"value":1355},"# Load optimized activation kernels\n",{"type":48,"tag":734,"props":1357,"children":1358},{"class":736,"line":764},[1359],{"type":48,"tag":734,"props":1360,"children":1361},{},[1362],{"type":54,"value":1363},"activation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n",{"type":48,"tag":734,"props":1365,"children":1366},{"class":736,"line":823},[1367],{"type":48,"tag":734,"props":1368,"children":1369},{"emptyLinePlaceholder":41},[1370],{"type":54,"value":761},{"type":48,"tag":734,"props":1372,"children":1373},{"class":736,"line":832},[1374],{"type":48,"tag":734,"props":1375,"children":1376},{},[1377],{"type":54,"value":1378},"# Use the kernel\n",{"type":48,"tag":734,"props":1380,"children":1381},{"class":736,"line":841},[1382],{"type":48,"tag":734,"props":1383,"children":1384},{},[1385],{"type":54,"value":1386},"y = torch.empty_like(x)\n",{"type":48,"tag":734,"props":1388,"children":1389},{"class":736,"line":850},[1390],{"type":48,"tag":734,"props":1391,"children":1392},{},[1393],{"type":54,"value":1394},"activation.gelu_fast(y, x)\n",{"type":48,"tag":57,"props":1396,"children":1397},{},[1398],{"type":48,"tag":63,"props":1399,"children":1400},{},[1401],{"type":54,"value":1402},"For a complete HuggingFace Kernels example:",{"type":48,"tag":723,"props":1404,"children":1406},{"className":1119,"code":1405,"language":1121,"meta":728,"style":728},"python scripts\u002Fhuggingface_kernels_example.py\n",[1407],{"type":48,"tag":171,"props":1408,"children":1409},{"__ignoreMap":728},[1410],{"type":48,"tag":734,"props":1411,"children":1412},{"class":736,"line":737},[1413,1417],{"type":48,"tag":734,"props":1414,"children":1415},{"style":1140},[1416],{"type":54,"value":23},{"type":48,"tag":734,"props":1418,"children":1419},{"style":1145},[1420],{"type":54,"value":1421}," scripts\u002Fhuggingface_kernels_example.py\n",{"type":48,"tag":123,"props":1423,"children":1425},{"id":1424},"isolated-kernel-micro-benchmarks",[1426],{"type":54,"value":1427},"Isolated Kernel Micro-benchmarks",{"type":48,"tag":723,"props":1429,"children":1431},{"className":1119,"code":1430,"language":1121,"meta":728,"style":728},"python benchmark_rmsnorm.py\n",[1432],{"type":48,"tag":171,"props":1433,"children":1434},{"__ignoreMap":728},[1435],{"type":48,"tag":734,"props":1436,"children":1437},{"class":736,"line":737},[1438,1442],{"type":48,"tag":734,"props":1439,"children":1440},{"style":1140},[1441],{"type":54,"value":23},{"type":48,"tag":734,"props":1443,"children":1444},{"style":1145},[1445],{"type":54,"value":1446}," benchmark_rmsnorm.py\n",{"type":48,"tag":78,"props":1448,"children":1450},{"id":1449},"supported-libraries-models",[1451],{"type":54,"value":1452},"Supported Libraries & Models",{"type":48,"tag":130,"props":1454,"children":1455},{},[1456,1477],{"type":48,"tag":134,"props":1457,"children":1458},{},[1459],{"type":48,"tag":138,"props":1460,"children":1461},{},[1462,1467,1472],{"type":48,"tag":142,"props":1463,"children":1464},{},[1465],{"type":54,"value":1466},"Library",{"type":48,"tag":142,"props":1468,"children":1469},{},[1470],{"type":54,"value":1471},"Supported Models",{"type":48,"tag":142,"props":1473,"children":1474},{},[1475],{"type":54,"value":1476},"Key Kernels",{"type":48,"tag":158,"props":1478,"children":1479},{},[1480,1500],{"type":48,"tag":138,"props":1481,"children":1482},{},[1483,1490,1495],{"type":48,"tag":165,"props":1484,"children":1485},{},[1486],{"type":48,"tag":63,"props":1487,"children":1488},{},[1489],{"type":54,"value":67},{"type":48,"tag":165,"props":1491,"children":1492},{},[1493],{"type":54,"value":1494},"LTX-Video, Stable Diffusion, FLUX, DiT",{"type":48,"tag":165,"props":1496,"children":1497},{},[1498],{"type":54,"value":1499},"RMSNorm, GEGLU, RoPE, AdaLN",{"type":48,"tag":138,"props":1501,"children":1502},{},[1503,1510,1515],{"type":48,"tag":165,"props":1504,"children":1505},{},[1506],{"type":48,"tag":63,"props":1507,"children":1508},{},[1509],{"type":54,"value":74},{"type":48,"tag":165,"props":1511,"children":1512},{},[1513],{"type":54,"value":1514},"LLaMA, Mistral, Qwen, Falcon",{"type":48,"tag":165,"props":1516,"children":1517},{},[1518],{"type":54,"value":1519},"RMSNorm, Attention",{"type":48,"tag":130,"props":1521,"children":1522},{},[1523,1544],{"type":48,"tag":134,"props":1524,"children":1525},{},[1526],{"type":48,"tag":138,"props":1527,"children":1528},{},[1529,1534,1539],{"type":48,"tag":142,"props":1530,"children":1531},{},[1532],{"type":54,"value":1533},"GPU",{"type":48,"tag":142,"props":1535,"children":1536},{},[1537],{"type":54,"value":1538},"Compute Capability",{"type":48,"tag":142,"props":1540,"children":1541},{},[1542],{"type":54,"value":1543},"Guide",{"type":48,"tag":158,"props":1545,"children":1546},{},[1547,1569,1591],{"type":48,"tag":138,"props":1548,"children":1549},{},[1550,1555,1560],{"type":48,"tag":165,"props":1551,"children":1552},{},[1553],{"type":54,"value":1554},"H100",{"type":48,"tag":165,"props":1556,"children":1557},{},[1558],{"type":54,"value":1559},"sm_90",{"type":48,"tag":165,"props":1561,"children":1562},{},[1563],{"type":48,"tag":90,"props":1564,"children":1566},{"href":1565},"references\u002Fh100-optimization-guide.md",[1567],{"type":54,"value":1568},"h100-optimization-guide.md",{"type":48,"tag":138,"props":1570,"children":1571},{},[1572,1577,1582],{"type":48,"tag":165,"props":1573,"children":1574},{},[1575],{"type":54,"value":1576},"A100",{"type":48,"tag":165,"props":1578,"children":1579},{},[1580],{"type":54,"value":1581},"sm_80",{"type":48,"tag":165,"props":1583,"children":1584},{},[1585],{"type":48,"tag":90,"props":1586,"children":1588},{"href":1587},"references\u002Fa100-optimization-guide.md",[1589],{"type":54,"value":1590},"a100-optimization-guide.md",{"type":48,"tag":138,"props":1592,"children":1593},{},[1594,1599,1604],{"type":48,"tag":165,"props":1595,"children":1596},{},[1597],{"type":54,"value":1598},"T4",{"type":48,"tag":165,"props":1600,"children":1601},{},[1602],{"type":54,"value":1603},"sm_75",{"type":48,"tag":165,"props":1605,"children":1606},{},[1607],{"type":48,"tag":90,"props":1608,"children":1610},{"href":1609},"references\u002Ft4-optimization-guide.md",[1611],{"type":54,"value":1612},"t4-optimization-guide.md",{"type":48,"tag":78,"props":1614,"children":1616},{"id":1615},"when-this-skill-applies",[1617],{"type":54,"value":1618},"When This Skill Applies",{"type":48,"tag":57,"props":1620,"children":1621},{},[1622],{"type":54,"value":1623},"Use this skill when:",{"type":48,"tag":1625,"props":1626,"children":1627},"ul",{},[1628,1638,1643,1648,1653,1664,1674],{"type":48,"tag":961,"props":1629,"children":1630},{},[1631,1636],{"type":48,"tag":63,"props":1632,"children":1633},{},[1634],{"type":54,"value":1635},"Benchmarking kernel performance",{"type":54,"value":1637}," against baseline implementations",{"type":48,"tag":961,"props":1639,"children":1640},{},[1641],{"type":54,"value":1642},"Writing new CUDA kernels for diffusion models or LLMs",{"type":48,"tag":961,"props":1644,"children":1645},{},[1646],{"type":54,"value":1647},"Optimizing existing kernels for H100, A100, or T4 architecture",{"type":48,"tag":961,"props":1649,"children":1650},{},[1651],{"type":54,"value":1652},"Implementing custom attention, normalization, or activation layers",{"type":48,"tag":961,"props":1654,"children":1655},{},[1656,1658,1662],{"type":54,"value":1657},"Integrating kernels with ",{"type":48,"tag":63,"props":1659,"children":1660},{},[1661],{"type":54,"value":67},{"type":54,"value":1663}," pipelines (LTX-Video, Stable Diffusion, FLUX, DiT)",{"type":48,"tag":961,"props":1665,"children":1666},{},[1667,1668,1672],{"type":54,"value":1657},{"type":48,"tag":63,"props":1669,"children":1670},{},[1671],{"type":54,"value":74},{"type":54,"value":1673}," models (LLaMA, Mistral, Qwen)",{"type":48,"tag":961,"props":1675,"children":1676},{},[1677],{"type":54,"value":1678},"Debugging kernel performance issues on NVIDIA GPUs",{"type":48,"tag":78,"props":1680,"children":1682},{"id":1681},"working-example",[1683],{"type":54,"value":1684},"Working Example",{"type":48,"tag":57,"props":1686,"children":1687},{},[1688,1690,1696,1698,1705],{"type":54,"value":1689},"Complete working examples ship with the kernels repo under ",{"type":48,"tag":171,"props":1691,"children":1693},{"className":1692},[],[1694],{"type":54,"value":1695},"examples\u002Fkernels\u002F",{"type":54,"value":1697}," (also at ",{"type":48,"tag":90,"props":1699,"children":1702},{"href":1700,"rel":1701},"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002Fmain\u002Fexamples\u002Fkernels",[93],[1703],{"type":54,"value":1704},"github.com\u002Fhuggingface\u002Fkernels",{"type":54,"value":1706},"):",{"type":48,"tag":1625,"props":1708,"children":1709},{},[1710,1736,1755],{"type":48,"tag":961,"props":1711,"children":1712},{},[1713,1719,1721,1726,1728,1734],{"type":48,"tag":171,"props":1714,"children":1716},{"className":1715},[],[1717],{"type":54,"value":1718},"relu\u002F",{"type":54,"value":1720}," — the canonical minimal kernel: build.toml, flake.nix, ",{"type":48,"tag":171,"props":1722,"children":1724},{"className":1723},[],[1725],{"type":54,"value":222},{"type":54,"value":1727}," bindings, Python API, ",{"type":48,"tag":171,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":54,"value":1733},"layers\u002F",{"type":54,"value":1735},", tests",{"type":48,"tag":961,"props":1737,"children":1738},{},[1739,1745,1747,1753],{"type":48,"tag":171,"props":1740,"children":1742},{"className":1741},[],[1743],{"type":54,"value":1744},"relu-backprop-compile\u002F",{"type":54,"value":1746}," — backward pass + ",{"type":48,"tag":171,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":54,"value":1752},"torch.compile",{"type":54,"value":1754}," support (fake-op registration)",{"type":48,"tag":961,"props":1756,"children":1757},{},[1758,1764],{"type":48,"tag":171,"props":1759,"children":1761},{"className":1760},[],[1762],{"type":54,"value":1763},"silu-and-mul\u002F",{"type":54,"value":1765}," — activation kernel following the same layout",{"type":48,"tag":78,"props":1767,"children":1769},{"id":1768},"benchmarking-kernels",[1770],{"type":54,"value":1771},"Benchmarking Kernels",{"type":48,"tag":57,"props":1773,"children":1774},{},[1775],{"type":54,"value":1776},"Use the benchmark script to measure kernel performance:",{"type":48,"tag":723,"props":1778,"children":1780},{"className":1119,"code":1779,"language":1121,"meta":728,"style":728},"# Full benchmark with all options\npython scripts\u002Fbenchmark_example.py \\\n    --use-optimized-kernels \\\n    --compile \\\n    --batch-size 1 \\\n    --num-frames 161 \\\n    --height 512 \\\n    --width 768 \\\n    --steps 50 \\\n    --warmup-iterations 2\n",[1781],{"type":48,"tag":171,"props":1782,"children":1783},{"__ignoreMap":728},[1784,1792,1808,1820,1832,1850,1867,1884,1901,1918],{"type":48,"tag":734,"props":1785,"children":1786},{"class":736,"line":737},[1787],{"type":48,"tag":734,"props":1788,"children":1789},{"style":1131},[1790],{"type":54,"value":1791},"# Full benchmark with all options\n",{"type":48,"tag":734,"props":1793,"children":1794},{"class":736,"line":746},[1795,1799,1804],{"type":48,"tag":734,"props":1796,"children":1797},{"style":1140},[1798],{"type":54,"value":23},{"type":48,"tag":734,"props":1800,"children":1801},{"style":1145},[1802],{"type":54,"value":1803}," scripts\u002Fbenchmark_example.py",{"type":48,"tag":734,"props":1805,"children":1806},{"style":1229},[1807],{"type":54,"value":1232},{"type":48,"tag":734,"props":1809,"children":1810},{"class":736,"line":755},[1811,1816],{"type":48,"tag":734,"props":1812,"children":1813},{"style":1145},[1814],{"type":54,"value":1815},"    --use-optimized-kernels",{"type":48,"tag":734,"props":1817,"children":1818},{"style":1229},[1819],{"type":54,"value":1232},{"type":48,"tag":734,"props":1821,"children":1822},{"class":736,"line":764},[1823,1828],{"type":48,"tag":734,"props":1824,"children":1825},{"style":1145},[1826],{"type":54,"value":1827},"    --compile",{"type":48,"tag":734,"props":1829,"children":1830},{"style":1229},[1831],{"type":54,"value":1232},{"type":48,"tag":734,"props":1833,"children":1834},{"class":736,"line":823},[1835,1840,1846],{"type":48,"tag":734,"props":1836,"children":1837},{"style":1145},[1838],{"type":54,"value":1839},"    --batch-size",{"type":48,"tag":734,"props":1841,"children":1843},{"style":1842},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1844],{"type":54,"value":1845}," 1",{"type":48,"tag":734,"props":1847,"children":1848},{"style":1229},[1849],{"type":54,"value":1232},{"type":48,"tag":734,"props":1851,"children":1852},{"class":736,"line":832},[1853,1858,1863],{"type":48,"tag":734,"props":1854,"children":1855},{"style":1145},[1856],{"type":54,"value":1857},"    --num-frames",{"type":48,"tag":734,"props":1859,"children":1860},{"style":1842},[1861],{"type":54,"value":1862}," 161",{"type":48,"tag":734,"props":1864,"children":1865},{"style":1229},[1866],{"type":54,"value":1232},{"type":48,"tag":734,"props":1868,"children":1869},{"class":736,"line":841},[1870,1875,1880],{"type":48,"tag":734,"props":1871,"children":1872},{"style":1145},[1873],{"type":54,"value":1874},"    --height",{"type":48,"tag":734,"props":1876,"children":1877},{"style":1842},[1878],{"type":54,"value":1879}," 512",{"type":48,"tag":734,"props":1881,"children":1882},{"style":1229},[1883],{"type":54,"value":1232},{"type":48,"tag":734,"props":1885,"children":1886},{"class":736,"line":850},[1887,1892,1897],{"type":48,"tag":734,"props":1888,"children":1889},{"style":1145},[1890],{"type":54,"value":1891},"    --width",{"type":48,"tag":734,"props":1893,"children":1894},{"style":1842},[1895],{"type":54,"value":1896}," 768",{"type":48,"tag":734,"props":1898,"children":1899},{"style":1229},[1900],{"type":54,"value":1232},{"type":48,"tag":734,"props":1902,"children":1903},{"class":736,"line":859},[1904,1909,1914],{"type":48,"tag":734,"props":1905,"children":1906},{"style":1145},[1907],{"type":54,"value":1908},"    --steps",{"type":48,"tag":734,"props":1910,"children":1911},{"style":1842},[1912],{"type":54,"value":1913}," 50",{"type":48,"tag":734,"props":1915,"children":1916},{"style":1229},[1917],{"type":54,"value":1232},{"type":48,"tag":734,"props":1919,"children":1920},{"class":736,"line":867},[1921,1926],{"type":48,"tag":734,"props":1922,"children":1923},{"style":1145},[1924],{"type":54,"value":1925},"    --warmup-iterations",{"type":48,"tag":734,"props":1927,"children":1928},{"style":1842},[1929],{"type":54,"value":1930}," 2\n",{"type":48,"tag":123,"props":1932,"children":1934},{"id":1933},"benchmark-script-options",[1935],{"type":54,"value":1936},"Benchmark Script Options",{"type":48,"tag":130,"props":1938,"children":1939},{},[1940,1961],{"type":48,"tag":134,"props":1941,"children":1942},{},[1943],{"type":48,"tag":138,"props":1944,"children":1945},{},[1946,1951,1956],{"type":48,"tag":142,"props":1947,"children":1948},{},[1949],{"type":54,"value":1950},"Option",{"type":48,"tag":142,"props":1952,"children":1953},{},[1954],{"type":54,"value":1955},"Default",{"type":48,"tag":142,"props":1957,"children":1958},{},[1959],{"type":54,"value":1960},"Description",{"type":48,"tag":158,"props":1962,"children":1963},{},[1964,1986,2008,2030,2052,2074,2096,2118,2140],{"type":48,"tag":138,"props":1965,"children":1966},{},[1967,1976,1981],{"type":48,"tag":165,"props":1968,"children":1969},{},[1970],{"type":48,"tag":171,"props":1971,"children":1973},{"className":1972},[],[1974],{"type":54,"value":1975},"--use-optimized-kernels",{"type":48,"tag":165,"props":1977,"children":1978},{},[1979],{"type":54,"value":1980},"auto",{"type":48,"tag":165,"props":1982,"children":1983},{},[1984],{"type":54,"value":1985},"Use custom H100 CUDA kernels",{"type":48,"tag":138,"props":1987,"children":1988},{},[1989,1998,2003],{"type":48,"tag":165,"props":1990,"children":1991},{},[1992],{"type":48,"tag":171,"props":1993,"children":1995},{"className":1994},[],[1996],{"type":54,"value":1997},"--no-optimized-kernels",{"type":48,"tag":165,"props":1999,"children":2000},{},[2001],{"type":54,"value":2002},"-",{"type":48,"tag":165,"props":2004,"children":2005},{},[2006],{"type":54,"value":2007},"Use baseline implementation",{"type":48,"tag":138,"props":2009,"children":2010},{},[2011,2020,2025],{"type":48,"tag":165,"props":2012,"children":2013},{},[2014],{"type":48,"tag":171,"props":2015,"children":2017},{"className":2016},[],[2018],{"type":54,"value":2019},"--compile",{"type":48,"tag":165,"props":2021,"children":2022},{},[2023],{"type":54,"value":2024},"false",{"type":48,"tag":165,"props":2026,"children":2027},{},[2028],{"type":54,"value":2029},"Enable torch.compile on transformer",{"type":48,"tag":138,"props":2031,"children":2032},{},[2033,2042,2047],{"type":48,"tag":165,"props":2034,"children":2035},{},[2036],{"type":48,"tag":171,"props":2037,"children":2039},{"className":2038},[],[2040],{"type":54,"value":2041},"--batch-size",{"type":48,"tag":165,"props":2043,"children":2044},{},[2045],{"type":54,"value":2046},"1",{"type":48,"tag":165,"props":2048,"children":2049},{},[2050],{"type":54,"value":2051},"Number of videos per prompt",{"type":48,"tag":138,"props":2053,"children":2054},{},[2055,2064,2069],{"type":48,"tag":165,"props":2056,"children":2057},{},[2058],{"type":48,"tag":171,"props":2059,"children":2061},{"className":2060},[],[2062],{"type":54,"value":2063},"--num-frames",{"type":48,"tag":165,"props":2065,"children":2066},{},[2067],{"type":54,"value":2068},"161",{"type":48,"tag":165,"props":2070,"children":2071},{},[2072],{"type":54,"value":2073},"Number of frames to generate",{"type":48,"tag":138,"props":2075,"children":2076},{},[2077,2086,2091],{"type":48,"tag":165,"props":2078,"children":2079},{},[2080],{"type":48,"tag":171,"props":2081,"children":2083},{"className":2082},[],[2084],{"type":54,"value":2085},"--height",{"type":48,"tag":165,"props":2087,"children":2088},{},[2089],{"type":54,"value":2090},"512",{"type":48,"tag":165,"props":2092,"children":2093},{},[2094],{"type":54,"value":2095},"Video height in pixels",{"type":48,"tag":138,"props":2097,"children":2098},{},[2099,2108,2113],{"type":48,"tag":165,"props":2100,"children":2101},{},[2102],{"type":48,"tag":171,"props":2103,"children":2105},{"className":2104},[],[2106],{"type":54,"value":2107},"--width",{"type":48,"tag":165,"props":2109,"children":2110},{},[2111],{"type":54,"value":2112},"768",{"type":48,"tag":165,"props":2114,"children":2115},{},[2116],{"type":54,"value":2117},"Video width in pixels",{"type":48,"tag":138,"props":2119,"children":2120},{},[2121,2130,2135],{"type":48,"tag":165,"props":2122,"children":2123},{},[2124],{"type":48,"tag":171,"props":2125,"children":2127},{"className":2126},[],[2128],{"type":54,"value":2129},"--steps",{"type":48,"tag":165,"props":2131,"children":2132},{},[2133],{"type":54,"value":2134},"50",{"type":48,"tag":165,"props":2136,"children":2137},{},[2138],{"type":54,"value":2139},"Denoising steps",{"type":48,"tag":138,"props":2141,"children":2142},{},[2143,2152,2157],{"type":48,"tag":165,"props":2144,"children":2145},{},[2146],{"type":48,"tag":171,"props":2147,"children":2149},{"className":2148},[],[2150],{"type":54,"value":2151},"--warmup-iterations",{"type":48,"tag":165,"props":2153,"children":2154},{},[2155],{"type":54,"value":2156},"2",{"type":48,"tag":165,"props":2158,"children":2159},{},[2160],{"type":54,"value":2161},"Warmup runs before benchmark",{"type":48,"tag":123,"props":2163,"children":2165},{"id":2164},"example-benchmark-results",[2166],{"type":54,"value":2167},"Example Benchmark Results",{"type":48,"tag":57,"props":2169,"children":2170},{},[2171],{"type":48,"tag":63,"props":2172,"children":2173},{},[2174],{"type":54,"value":2175},"End-to-End Video Generation (49 frames, 30 steps, H100 80GB):",{"type":48,"tag":130,"props":2177,"children":2178},{},[2179,2212],{"type":48,"tag":134,"props":2180,"children":2181},{},[2182],{"type":48,"tag":138,"props":2183,"children":2184},{},[2185,2191,2197,2202,2207],{"type":48,"tag":142,"props":2186,"children":2188},{"align":2187},"left",[2189],{"type":54,"value":2190},"Configuration",{"type":48,"tag":142,"props":2192,"children":2194},{"align":2193},"center",[2195],{"type":54,"value":2196},"Time (s)",{"type":48,"tag":142,"props":2198,"children":2199},{"align":2193},[2200],{"type":54,"value":2201},"it\u002Fs",{"type":48,"tag":142,"props":2203,"children":2204},{"align":2193},[2205],{"type":54,"value":2206},"Speedup",{"type":48,"tag":142,"props":2208,"children":2209},{"align":2187},[2210],{"type":54,"value":2211},"Notes",{"type":48,"tag":158,"props":2213,"children":2214},{},[2215,2243,2277],{"type":48,"tag":138,"props":2216,"children":2217},{},[2218,2223,2228,2233,2238],{"type":48,"tag":165,"props":2219,"children":2220},{"align":2187},[2221],{"type":54,"value":2222},"Baseline (no compile)",{"type":48,"tag":165,"props":2224,"children":2225},{"align":2193},[2226],{"type":54,"value":2227},"2.87",{"type":48,"tag":165,"props":2229,"children":2230},{"align":2193},[2231],{"type":54,"value":2232},"12.58",{"type":48,"tag":165,"props":2234,"children":2235},{"align":2193},[2236],{"type":54,"value":2237},"1.00x",{"type":48,"tag":165,"props":2239,"children":2240},{"align":2187},[2241],{"type":54,"value":2242},"Reference",{"type":48,"tag":138,"props":2244,"children":2245},{},[2246,2254,2259,2264,2272],{"type":48,"tag":165,"props":2247,"children":2248},{"align":2187},[2249],{"type":48,"tag":63,"props":2250,"children":2251},{},[2252],{"type":54,"value":2253},"Optimized Kernels",{"type":48,"tag":165,"props":2255,"children":2256},{"align":2193},[2257],{"type":54,"value":2258},"2.70",{"type":48,"tag":165,"props":2260,"children":2261},{"align":2193},[2262],{"type":54,"value":2263},"13.52",{"type":48,"tag":165,"props":2265,"children":2266},{"align":2193},[2267],{"type":48,"tag":63,"props":2268,"children":2269},{},[2270],{"type":54,"value":2271},"1.06x",{"type":48,"tag":165,"props":2273,"children":2274},{"align":2187},[2275],{"type":54,"value":2276},"6% faster",{"type":48,"tag":138,"props":2278,"children":2279},{},[2280,2285,2290,2295,2300],{"type":48,"tag":165,"props":2281,"children":2282},{"align":2187},[2283],{"type":54,"value":2284},"Baseline + torch.compile",{"type":48,"tag":165,"props":2286,"children":2287},{"align":2193},[2288],{"type":54,"value":2289},"2.14",{"type":48,"tag":165,"props":2291,"children":2292},{"align":2193},[2293],{"type":54,"value":2294},"19.05",{"type":48,"tag":165,"props":2296,"children":2297},{"align":2193},[2298],{"type":54,"value":2299},"1.34x",{"type":48,"tag":165,"props":2301,"children":2302},{"align":2187},[2303],{"type":54,"value":2304},"34% faster",{"type":48,"tag":57,"props":2306,"children":2307},{},[2308,2313,2314,2319,2320,2325],{"type":48,"tag":63,"props":2309,"children":2310},{},[2311],{"type":54,"value":2312},"Important:",{"type":54,"value":1035},{"type":48,"tag":171,"props":2315,"children":2317},{"className":2316},[],[2318],{"type":54,"value":1975},{"type":54,"value":69},{"type":48,"tag":171,"props":2321,"children":2323},{"className":2322},[],[2324],{"type":54,"value":2019},{"type":54,"value":2326}," are currently mutually exclusive. Custom kernels require PyTorch custom op registration to work with torch.compile.",{"type":48,"tag":57,"props":2328,"children":2329},{},[2330],{"type":48,"tag":63,"props":2331,"children":2332},{},[2333],{"type":54,"value":2334},"Key metrics to capture:",{"type":48,"tag":1625,"props":2336,"children":2337},{},[2338,2348,2358,2368],{"type":48,"tag":961,"props":2339,"children":2340},{},[2341,2346],{"type":48,"tag":63,"props":2342,"children":2343},{},[2344],{"type":54,"value":2345},"Device:",{"type":54,"value":2347}," GPU model (e.g., NVIDIA H100 80GB HBM3)",{"type":48,"tag":961,"props":2349,"children":2350},{},[2351,2356],{"type":48,"tag":63,"props":2352,"children":2353},{},[2354],{"type":54,"value":2355},"Precision:",{"type":54,"value":2357}," Data type used (e.g., bfloat16)",{"type":48,"tag":961,"props":2359,"children":2360},{},[2361,2366],{"type":48,"tag":63,"props":2362,"children":2363},{},[2364],{"type":54,"value":2365},"Resolution:",{"type":54,"value":2367}," Width x Height (e.g., 768x512)",{"type":48,"tag":961,"props":2369,"children":2370},{},[2371,2376],{"type":48,"tag":63,"props":2372,"children":2373},{},[2374],{"type":54,"value":2375},"Frames:",{"type":54,"value":2377}," Number of frames generated (e.g., 49, 161)",{"type":48,"tag":123,"props":2379,"children":2381},{"id":2380},"rmsnorm-micro-benchmarks",[2382],{"type":54,"value":2383},"RMSNorm Micro-benchmarks",{"type":48,"tag":57,"props":2385,"children":2386},{},[2387,2389,2394],{"type":54,"value":2388},"The vectorized RMSNorm kernel achieves ",{"type":48,"tag":63,"props":2390,"children":2391},{},[2392],{"type":54,"value":2393},"2.67x average speedup",{"type":54,"value":2395}," over PyTorch baseline:",{"type":48,"tag":130,"props":2397,"children":2398},{},[2399,2424],{"type":48,"tag":134,"props":2400,"children":2401},{},[2402],{"type":48,"tag":138,"props":2403,"children":2404},{},[2405,2410,2415,2420],{"type":48,"tag":142,"props":2406,"children":2407},{"align":2187},[2408],{"type":54,"value":2409},"Shape",{"type":48,"tag":142,"props":2411,"children":2412},{"align":2193},[2413],{"type":54,"value":2414},"Custom (ms)",{"type":48,"tag":142,"props":2416,"children":2417},{"align":2193},[2418],{"type":54,"value":2419},"PyTorch (ms)",{"type":48,"tag":142,"props":2421,"children":2422},{"align":2193},[2423],{"type":54,"value":2206},{"type":48,"tag":158,"props":2425,"children":2426},{},[2427,2456,2485,2514,2543],{"type":48,"tag":138,"props":2428,"children":2429},{},[2430,2438,2443,2448],{"type":48,"tag":165,"props":2431,"children":2432},{"align":2187},[2433],{"type":48,"tag":734,"props":2434,"children":2435},{},[2436],{"type":54,"value":2437},"1×1024×2048",{"type":48,"tag":165,"props":2439,"children":2440},{"align":2193},[2441],{"type":54,"value":2442},"0.019",{"type":48,"tag":165,"props":2444,"children":2445},{"align":2193},[2446],{"type":54,"value":2447},"0.065",{"type":48,"tag":165,"props":2449,"children":2450},{"align":2193},[2451],{"type":48,"tag":63,"props":2452,"children":2453},{},[2454],{"type":54,"value":2455},"3.37x",{"type":48,"tag":138,"props":2457,"children":2458},{},[2459,2467,2472,2477],{"type":48,"tag":165,"props":2460,"children":2461},{"align":2187},[2462],{"type":48,"tag":734,"props":2463,"children":2464},{},[2465],{"type":54,"value":2466},"2×1024×2048",{"type":48,"tag":165,"props":2468,"children":2469},{"align":2193},[2470],{"type":54,"value":2471},"0.024",{"type":48,"tag":165,"props":2473,"children":2474},{"align":2193},[2475],{"type":54,"value":2476},"0.073",{"type":48,"tag":165,"props":2478,"children":2479},{"align":2193},[2480],{"type":48,"tag":63,"props":2481,"children":2482},{},[2483],{"type":54,"value":2484},"3.04x",{"type":48,"tag":138,"props":2486,"children":2487},{},[2488,2496,2501,2506],{"type":48,"tag":165,"props":2489,"children":2490},{"align":2187},[2491],{"type":48,"tag":734,"props":2492,"children":2493},{},[2494],{"type":54,"value":2495},"4×1024×2048",{"type":48,"tag":165,"props":2497,"children":2498},{"align":2193},[2499],{"type":54,"value":2500},"0.036",{"type":48,"tag":165,"props":2502,"children":2503},{"align":2193},[2504],{"type":54,"value":2505},"0.093",{"type":48,"tag":165,"props":2507,"children":2508},{"align":2193},[2509],{"type":48,"tag":63,"props":2510,"children":2511},{},[2512],{"type":54,"value":2513},"2.58x",{"type":48,"tag":138,"props":2515,"children":2516},{},[2517,2525,2530,2535],{"type":48,"tag":165,"props":2518,"children":2519},{"align":2187},[2520],{"type":48,"tag":734,"props":2521,"children":2522},{},[2523],{"type":54,"value":2524},"2×4096×3072",{"type":48,"tag":165,"props":2526,"children":2527},{"align":2193},[2528],{"type":54,"value":2529},"0.087",{"type":48,"tag":165,"props":2531,"children":2532},{"align":2193},[2533],{"type":54,"value":2534},"0.208",{"type":48,"tag":165,"props":2536,"children":2537},{"align":2193},[2538],{"type":48,"tag":63,"props":2539,"children":2540},{},[2541],{"type":54,"value":2542},"2.41x",{"type":48,"tag":138,"props":2544,"children":2545},{},[2546,2554,2559,2564],{"type":48,"tag":165,"props":2547,"children":2548},{"align":2187},[2549],{"type":48,"tag":734,"props":2550,"children":2551},{},[2552],{"type":54,"value":2553},"4×4096×3072",{"type":48,"tag":165,"props":2555,"children":2556},{"align":2193},[2557],{"type":54,"value":2558},"0.157",{"type":48,"tag":165,"props":2560,"children":2561},{"align":2193},[2562],{"type":54,"value":2563},"0.392",{"type":48,"tag":165,"props":2565,"children":2566},{"align":2193},[2567],{"type":48,"tag":63,"props":2568,"children":2569},{},[2570],{"type":54,"value":2571},"2.49x",{"type":48,"tag":57,"props":2573,"children":2574},{},[2575,2580],{"type":48,"tag":63,"props":2576,"children":2577},{},[2578],{"type":54,"value":2579},"Bandwidth efficiency:",{"type":54,"value":2581}," 38% of H100's theoretical 3.35 TB\u002Fs",{"type":48,"tag":57,"props":2583,"children":2584},{},[2585,2590],{"type":48,"tag":63,"props":2586,"children":2587},{},[2588],{"type":54,"value":2589},"Why end-to-end speedup is smaller:",{"type":54,"value":2591}," RMSNorm accounts for ~5% of total compute in LTX-Video. The remaining time is spent in attention (Flash Attention\u002FSDPA), linear projections, and VAE decode.",{"type":48,"tag":78,"props":2593,"children":2595},{"id":2594},"project-structure",[2596],{"type":54,"value":2597},"Project Structure",{"type":48,"tag":723,"props":2599,"children":2603},{"className":2600,"code":2602,"language":54},[2601],"language-text",".claude\u002Fskills\u002Fcuda-kernels\u002F\n├── scripts\u002F\n│   ├── benchmark_example.py              # End-to-end video generation benchmark\n│   ├── benchmark_rmsnorm.py              # Isolated RMSNorm micro-benchmark\n│   ├── ltx_kernel_injection_example.py   # Minimal diffusers integration (~150 lines)\n│   ├── transformers_injection_example.py # Minimal transformers integration (~120 lines)\n│   └── huggingface_kernels_example.py    # HuggingFace Kernels Hub integration\n├── references\u002F\n│   ├── diffusers-integration.md          # Complete diffusers integration guide\n│   ├── transformers-integration.md       # Complete transformers integration guide\n│   ├── huggingface-kernels-integration.md # HuggingFace Kernels Hub (get_kernel) guide\n│   ├── troubleshooting.md                # Common issues and solutions\n│   ├── kernel-templates.md               # CUDA kernel templates (includes vectorized)\n│   ├── h100-optimization-guide.md        # H100 (Hopper) optimization deep dive\n│   ├── a100-optimization-guide.md        # A100 (Ampere) optimization deep dive\n│   └── t4-optimization-guide.md          # T4 (Turing) optimization deep dive\n└── SKILL.md                              # This file\n\nexamples\u002Fkernels\u002Frelu\u002F               # Canonical working example (kernels repo)\n├── build.toml                      # kernel-builder build configuration\n├── flake.nix                       # Nix build entry point\n├── CARD.md                         # Kernel card template (becomes README.md)\n├── relu_cuda\u002Frelu.cu               # CUDA kernel source\n├── torch-ext\u002F\n│   ├── torch_binding.h \u002F .cpp      # TORCH_LIBRARY_EXPAND bindings\n│   └── relu\u002F__init__.py            # Python API (+ optional layers\u002F)\n└── tests\u002Ftest_relu.py              # Kernel tests (nix run .#ci-test)\n",[2604],{"type":48,"tag":171,"props":2605,"children":2606},{"__ignoreMap":728},[2607],{"type":54,"value":2602},{"type":48,"tag":78,"props":2609,"children":2611},{"id":2610},"gpu-architecture-reference",[2612],{"type":54,"value":2613},"GPU Architecture Reference",{"type":48,"tag":123,"props":2615,"children":2617},{"id":2616},"h100-hopper-primary-target",[2618],{"type":54,"value":2619},"H100 (Hopper) - Primary Target",{"type":48,"tag":130,"props":2621,"children":2622},{},[2623,2644],{"type":48,"tag":134,"props":2624,"children":2625},{},[2626],{"type":48,"tag":138,"props":2627,"children":2628},{},[2629,2634,2639],{"type":48,"tag":142,"props":2630,"children":2631},{},[2632],{"type":54,"value":2633},"Spec",{"type":48,"tag":142,"props":2635,"children":2636},{},[2637],{"type":54,"value":2638},"Value",{"type":48,"tag":142,"props":2640,"children":2641},{},[2642],{"type":54,"value":2643},"Optimization Impact",{"type":48,"tag":158,"props":2645,"children":2646},{},[2647,2665,2683,2701,2719,2737],{"type":48,"tag":138,"props":2648,"children":2649},{},[2650,2655,2660],{"type":48,"tag":165,"props":2651,"children":2652},{},[2653],{"type":54,"value":2654},"SMs",{"type":48,"tag":165,"props":2656,"children":2657},{},[2658],{"type":54,"value":2659},"132",{"type":48,"tag":165,"props":2661,"children":2662},{},[2663],{"type":54,"value":2664},"Grid sizing: aim for multiples of 132",{"type":48,"tag":138,"props":2666,"children":2667},{},[2668,2673,2678],{"type":48,"tag":165,"props":2669,"children":2670},{},[2671],{"type":54,"value":2672},"Threads\u002FSM",{"type":48,"tag":165,"props":2674,"children":2675},{},[2676],{"type":54,"value":2677},"2048",{"type":48,"tag":165,"props":2679,"children":2680},{},[2681],{"type":54,"value":2682},"Max 16 blocks of 128 threads per SM",{"type":48,"tag":138,"props":2684,"children":2685},{},[2686,2691,2696],{"type":48,"tag":165,"props":2687,"children":2688},{},[2689],{"type":54,"value":2690},"Shared Memory",{"type":48,"tag":165,"props":2692,"children":2693},{},[2694],{"type":54,"value":2695},"192 KB\u002FSM",{"type":48,"tag":165,"props":2697,"children":2698},{},[2699],{"type":54,"value":2700},"Large tiles possible",{"type":48,"tag":138,"props":2702,"children":2703},{},[2704,2709,2714],{"type":48,"tag":165,"props":2705,"children":2706},{},[2707],{"type":54,"value":2708},"L2 Cache",{"type":48,"tag":165,"props":2710,"children":2711},{},[2712],{"type":54,"value":2713},"50 MB",{"type":48,"tag":165,"props":2715,"children":2716},{},[2717],{"type":54,"value":2718},"Reuse across blocks",{"type":48,"tag":138,"props":2720,"children":2721},{},[2722,2727,2732],{"type":48,"tag":165,"props":2723,"children":2724},{},[2725],{"type":54,"value":2726},"Memory BW",{"type":48,"tag":165,"props":2728,"children":2729},{},[2730],{"type":54,"value":2731},"3.35 TB\u002Fs",{"type":48,"tag":165,"props":2733,"children":2734},{},[2735],{"type":54,"value":2736},"Coalesced access critical",{"type":48,"tag":138,"props":2738,"children":2739},{},[2740,2745,2750],{"type":48,"tag":165,"props":2741,"children":2742},{},[2743],{"type":54,"value":2744},"Warp Size",{"type":48,"tag":165,"props":2746,"children":2747},{},[2748],{"type":54,"value":2749},"32",{"type":48,"tag":165,"props":2751,"children":2752},{},[2753],{"type":54,"value":2754},"All reductions use warp shuffles",{"type":48,"tag":123,"props":2756,"children":2758},{"id":2757},"quick-comparison-h100-vs-a100-vs-t4",[2759],{"type":54,"value":2760},"Quick Comparison (H100 vs A100 vs T4)",{"type":48,"tag":130,"props":2762,"children":2763},{},[2764,2786],{"type":48,"tag":134,"props":2765,"children":2766},{},[2767],{"type":48,"tag":138,"props":2768,"children":2769},{},[2770,2774,2778,2782],{"type":48,"tag":142,"props":2771,"children":2772},{},[2773],{"type":54,"value":2633},{"type":48,"tag":142,"props":2775,"children":2776},{},[2777],{"type":54,"value":1554},{"type":48,"tag":142,"props":2779,"children":2780},{},[2781],{"type":54,"value":1576},{"type":48,"tag":142,"props":2783,"children":2784},{},[2785],{"type":54,"value":1598},{"type":48,"tag":158,"props":2787,"children":2788},{},[2789,2810,2831,2854,2879],{"type":48,"tag":138,"props":2790,"children":2791},{},[2792,2796,2800,2805],{"type":48,"tag":165,"props":2793,"children":2794},{},[2795],{"type":54,"value":2654},{"type":48,"tag":165,"props":2797,"children":2798},{},[2799],{"type":54,"value":2659},{"type":48,"tag":165,"props":2801,"children":2802},{},[2803],{"type":54,"value":2804},"108",{"type":48,"tag":165,"props":2806,"children":2807},{},[2808],{"type":54,"value":2809},"40",{"type":48,"tag":138,"props":2811,"children":2812},{},[2813,2817,2821,2826],{"type":48,"tag":165,"props":2814,"children":2815},{},[2816],{"type":54,"value":2726},{"type":48,"tag":165,"props":2818,"children":2819},{},[2820],{"type":54,"value":2731},{"type":48,"tag":165,"props":2822,"children":2823},{},[2824],{"type":54,"value":2825},"2.0 TB\u002Fs",{"type":48,"tag":165,"props":2827,"children":2828},{},[2829],{"type":54,"value":2830},"320 GB\u002Fs",{"type":48,"tag":138,"props":2832,"children":2833},{},[2834,2839,2844,2849],{"type":48,"tag":165,"props":2835,"children":2836},{},[2837],{"type":54,"value":2838},"Shared Mem\u002FSM",{"type":48,"tag":165,"props":2840,"children":2841},{},[2842],{"type":54,"value":2843},"192 KB",{"type":48,"tag":165,"props":2845,"children":2846},{},[2847],{"type":54,"value":2848},"164 KB",{"type":48,"tag":165,"props":2850,"children":2851},{},[2852],{"type":54,"value":2853},"64 KB",{"type":48,"tag":138,"props":2855,"children":2856},{},[2857,2862,2867,2871],{"type":48,"tag":165,"props":2858,"children":2859},{},[2860],{"type":54,"value":2861},"BF16 Support",{"type":48,"tag":165,"props":2863,"children":2864},{},[2865],{"type":54,"value":2866},"Yes",{"type":48,"tag":165,"props":2868,"children":2869},{},[2870],{"type":54,"value":2866},{"type":48,"tag":165,"props":2872,"children":2873},{},[2874],{"type":48,"tag":63,"props":2875,"children":2876},{},[2877],{"type":54,"value":2878},"No (FP16 only)",{"type":48,"tag":138,"props":2880,"children":2881},{},[2882,2887,2891,2895],{"type":48,"tag":165,"props":2883,"children":2884},{},[2885],{"type":54,"value":2886},"Compute Cap",{"type":48,"tag":165,"props":2888,"children":2889},{},[2890],{"type":54,"value":1559},{"type":48,"tag":165,"props":2892,"children":2893},{},[2894],{"type":54,"value":1581},{"type":48,"tag":165,"props":2896,"children":2897},{},[2898],{"type":54,"value":1603},{"type":48,"tag":2900,"props":2901,"children":2902},"blockquote",{},[2903],{"type":48,"tag":57,"props":2904,"children":2905},{},[2906,2908,2912,2914,2918,2919],{"type":54,"value":2907},"See detailed guides: ",{"type":48,"tag":90,"props":2909,"children":2910},{"href":1565},[2911],{"type":54,"value":1554},{"type":54,"value":2913}," | ",{"type":48,"tag":90,"props":2915,"children":2916},{"href":1587},[2917],{"type":54,"value":1576},{"type":54,"value":2913},{"type":48,"tag":90,"props":2920,"children":2921},{"href":1609},[2922],{"type":54,"value":1598},{"type":48,"tag":78,"props":2924,"children":2926},{"id":2925},"core-kernel-patterns",[2927],{"type":54,"value":2928},"Core Kernel Patterns",{"type":48,"tag":123,"props":2930,"children":2932},{"id":2931},"vectorized-memory-access-critical-for-performance",[2933],{"type":54,"value":2934},"Vectorized Memory Access (Critical for Performance)",{"type":48,"tag":57,"props":2936,"children":2937},{},[2938],{"type":48,"tag":63,"props":2939,"children":2940},{},[2941,2943,2949],{"type":54,"value":2942},"BFloat16 vectorization using ",{"type":48,"tag":171,"props":2944,"children":2946},{"className":2945},[],[2947],{"type":54,"value":2948},"__nv_bfloat162",{"type":54,"value":721},{"type":48,"tag":723,"props":2951,"children":2955},{"className":2952,"code":2953,"language":2954,"meta":728,"style":728},"language-cuda shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\u002F\u002F Load 2 bfloat16 elements at once (32-bit load)\nconst __nv_bfloat162* vec_input = reinterpret_cast\u003Cconst __nv_bfloat162*>(row_input);\n\n#pragma unroll 4\nfor (int i = tid; i \u003C vec_hidden; i += stride) {\n    __nv_bfloat162 v = vec_input[i];\n    float v0 = __bfloat162float(v.x);\n    float v1 = __bfloat162float(v.y);\n    sum_sq += v0 * v0 + v1 * v1;\n}\n","cuda",[2956],{"type":48,"tag":171,"props":2957,"children":2958},{"__ignoreMap":728},[2959,2967,2975,2982,2990,2998,3006,3014,3022,3030],{"type":48,"tag":734,"props":2960,"children":2961},{"class":736,"line":737},[2962],{"type":48,"tag":734,"props":2963,"children":2964},{},[2965],{"type":54,"value":2966},"\u002F\u002F Load 2 bfloat16 elements at once (32-bit load)\n",{"type":48,"tag":734,"props":2968,"children":2969},{"class":736,"line":746},[2970],{"type":48,"tag":734,"props":2971,"children":2972},{},[2973],{"type":54,"value":2974},"const __nv_bfloat162* vec_input = reinterpret_cast\u003Cconst __nv_bfloat162*>(row_input);\n",{"type":48,"tag":734,"props":2976,"children":2977},{"class":736,"line":755},[2978],{"type":48,"tag":734,"props":2979,"children":2980},{"emptyLinePlaceholder":41},[2981],{"type":54,"value":761},{"type":48,"tag":734,"props":2983,"children":2984},{"class":736,"line":764},[2985],{"type":48,"tag":734,"props":2986,"children":2987},{},[2988],{"type":54,"value":2989},"#pragma unroll 4\n",{"type":48,"tag":734,"props":2991,"children":2992},{"class":736,"line":823},[2993],{"type":48,"tag":734,"props":2994,"children":2995},{},[2996],{"type":54,"value":2997},"for (int i = tid; i \u003C vec_hidden; i += stride) {\n",{"type":48,"tag":734,"props":2999,"children":3000},{"class":736,"line":832},[3001],{"type":48,"tag":734,"props":3002,"children":3003},{},[3004],{"type":54,"value":3005},"    __nv_bfloat162 v = vec_input[i];\n",{"type":48,"tag":734,"props":3007,"children":3008},{"class":736,"line":841},[3009],{"type":48,"tag":734,"props":3010,"children":3011},{},[3012],{"type":54,"value":3013},"    float v0 = __bfloat162float(v.x);\n",{"type":48,"tag":734,"props":3015,"children":3016},{"class":736,"line":850},[3017],{"type":48,"tag":734,"props":3018,"children":3019},{},[3020],{"type":54,"value":3021},"    float v1 = __bfloat162float(v.y);\n",{"type":48,"tag":734,"props":3023,"children":3024},{"class":736,"line":859},[3025],{"type":48,"tag":734,"props":3026,"children":3027},{},[3028],{"type":54,"value":3029},"    sum_sq += v0 * v0 + v1 * v1;\n",{"type":48,"tag":734,"props":3031,"children":3032},{"class":736,"line":867},[3033],{"type":48,"tag":734,"props":3034,"children":3035},{},[3036],{"type":54,"value":856},{"type":48,"tag":57,"props":3038,"children":3039},{},[3040],{"type":48,"tag":63,"props":3041,"children":3042},{},[3043,3045,3051],{"type":54,"value":3044},"FP16 vectorization using ",{"type":48,"tag":171,"props":3046,"children":3048},{"className":3047},[],[3049],{"type":54,"value":3050},"__half2",{"type":54,"value":721},{"type":48,"tag":723,"props":3053,"children":3055},{"className":2952,"code":3054,"language":2954,"meta":728,"style":728},"const __half2* vec_input = reinterpret_cast\u003Cconst __half2*>(row_input);\n__half2 v = vec_input[i];\nfloat v0 = __half2float(v.x);\nfloat v1 = __half2float(v.y);\n",[3056],{"type":48,"tag":171,"props":3057,"children":3058},{"__ignoreMap":728},[3059,3067,3075,3083],{"type":48,"tag":734,"props":3060,"children":3061},{"class":736,"line":737},[3062],{"type":48,"tag":734,"props":3063,"children":3064},{},[3065],{"type":54,"value":3066},"const __half2* vec_input = reinterpret_cast\u003Cconst __half2*>(row_input);\n",{"type":48,"tag":734,"props":3068,"children":3069},{"class":736,"line":746},[3070],{"type":48,"tag":734,"props":3071,"children":3072},{},[3073],{"type":54,"value":3074},"__half2 v = vec_input[i];\n",{"type":48,"tag":734,"props":3076,"children":3077},{"class":736,"line":755},[3078],{"type":48,"tag":734,"props":3079,"children":3080},{},[3081],{"type":54,"value":3082},"float v0 = __half2float(v.x);\n",{"type":48,"tag":734,"props":3084,"children":3085},{"class":736,"line":764},[3086],{"type":48,"tag":734,"props":3087,"children":3088},{},[3089],{"type":54,"value":3090},"float v1 = __half2float(v.y);\n",{"type":48,"tag":57,"props":3092,"children":3093},{},[3094],{"type":48,"tag":63,"props":3095,"children":3096},{},[3097,3099,3105],{"type":54,"value":3098},"FP32 vectorization using ",{"type":48,"tag":171,"props":3100,"children":3102},{"className":3101},[],[3103],{"type":54,"value":3104},"float4",{"type":54,"value":721},{"type":48,"tag":723,"props":3107,"children":3109},{"className":2952,"code":3108,"language":2954,"meta":728,"style":728},"const float4* vec_input = reinterpret_cast\u003Cconst float4*>(row_input);\nfloat4 v = vec_input[i];\nsum_sq += v.x * v.x + v.y * v.y + v.z * v.z + v.w * v.w;\n",[3110],{"type":48,"tag":171,"props":3111,"children":3112},{"__ignoreMap":728},[3113,3121,3129],{"type":48,"tag":734,"props":3114,"children":3115},{"class":736,"line":737},[3116],{"type":48,"tag":734,"props":3117,"children":3118},{},[3119],{"type":54,"value":3120},"const float4* vec_input = reinterpret_cast\u003Cconst float4*>(row_input);\n",{"type":48,"tag":734,"props":3122,"children":3123},{"class":736,"line":746},[3124],{"type":48,"tag":734,"props":3125,"children":3126},{},[3127],{"type":54,"value":3128},"float4 v = vec_input[i];\n",{"type":48,"tag":734,"props":3130,"children":3131},{"class":736,"line":755},[3132],{"type":48,"tag":734,"props":3133,"children":3134},{},[3135],{"type":54,"value":3136},"sum_sq += v.x * v.x + v.y * v.y + v.z * v.z + v.w * v.w;\n",{"type":48,"tag":123,"props":3138,"children":3140},{"id":3139},"warp-shuffle-reductions",[3141],{"type":54,"value":3142},"Warp Shuffle Reductions",{"type":48,"tag":723,"props":3144,"children":3146},{"className":2952,"code":3145,"language":2954,"meta":728,"style":728},"template \u003Ctypename T>\n__device__ __forceinline__ T warp_reduce_sum(T val) {\n    #pragma unroll\n    for (int offset = 16; offset > 0; offset >>= 1) {\n        val += __shfl_xor_sync(0xffffffff, val, offset);\n    }\n    return val;\n}\n",[3147],{"type":48,"tag":171,"props":3148,"children":3149},{"__ignoreMap":728},[3150,3158,3166,3174,3182,3190,3198,3206],{"type":48,"tag":734,"props":3151,"children":3152},{"class":736,"line":737},[3153],{"type":48,"tag":734,"props":3154,"children":3155},{},[3156],{"type":54,"value":3157},"template \u003Ctypename T>\n",{"type":48,"tag":734,"props":3159,"children":3160},{"class":736,"line":746},[3161],{"type":48,"tag":734,"props":3162,"children":3163},{},[3164],{"type":54,"value":3165},"__device__ __forceinline__ T warp_reduce_sum(T val) {\n",{"type":48,"tag":734,"props":3167,"children":3168},{"class":736,"line":755},[3169],{"type":48,"tag":734,"props":3170,"children":3171},{},[3172],{"type":54,"value":3173},"    #pragma unroll\n",{"type":48,"tag":734,"props":3175,"children":3176},{"class":736,"line":764},[3177],{"type":48,"tag":734,"props":3178,"children":3179},{},[3180],{"type":54,"value":3181},"    for (int offset = 16; offset > 0; offset >>= 1) {\n",{"type":48,"tag":734,"props":3183,"children":3184},{"class":736,"line":823},[3185],{"type":48,"tag":734,"props":3186,"children":3187},{},[3188],{"type":54,"value":3189},"        val += __shfl_xor_sync(0xffffffff, val, offset);\n",{"type":48,"tag":734,"props":3191,"children":3192},{"class":736,"line":832},[3193],{"type":48,"tag":734,"props":3194,"children":3195},{},[3196],{"type":54,"value":3197},"    }\n",{"type":48,"tag":734,"props":3199,"children":3200},{"class":736,"line":841},[3201],{"type":48,"tag":734,"props":3202,"children":3203},{},[3204],{"type":54,"value":3205},"    return val;\n",{"type":48,"tag":734,"props":3207,"children":3208},{"class":736,"line":850},[3209],{"type":48,"tag":734,"props":3210,"children":3211},{},[3212],{"type":54,"value":856},{"type":48,"tag":123,"props":3214,"children":3216},{"id":3215},"block-sizes-for-attention",[3217],{"type":54,"value":3218},"Block Sizes for Attention",{"type":48,"tag":1625,"props":3220,"children":3221},{},[3222,3245],{"type":48,"tag":961,"props":3223,"children":3224},{},[3225,3231,3232,3238,3239],{"type":48,"tag":171,"props":3226,"children":3228},{"className":3227},[],[3229],{"type":54,"value":3230},"BLOCK_SIZE_M = 128",{"type":54,"value":178},{"type":48,"tag":171,"props":3233,"children":3235},{"className":3234},[],[3236],{"type":54,"value":3237},"BLOCK_SIZE_N = 64",{"type":54,"value":178},{"type":48,"tag":171,"props":3240,"children":3242},{"className":3241},[],[3243],{"type":54,"value":3244},"BLOCK_SIZE_K = 64",{"type":48,"tag":961,"props":3246,"children":3247},{},[3248],{"type":48,"tag":171,"props":3249,"children":3251},{"className":3250},[],[3252],{"type":54,"value":3253},"NUM_WARPS = 8",{"type":48,"tag":123,"props":3255,"children":3257},{"id":3256},"thread-configuration",[3258],{"type":54,"value":3259},"Thread Configuration",{"type":48,"tag":57,"props":3261,"children":3262},{},[3263],{"type":54,"value":3264},"For element-wise ops (RoPE, GEGLU):",{"type":48,"tag":723,"props":3266,"children":3268},{"className":2952,"code":3267,"language":2954,"meta":728,"style":728},"constexpr int BLOCK_SIZE = 256;\nint num_blocks = (total_elements + BLOCK_SIZE - 1) \u002F BLOCK_SIZE;\n",[3269],{"type":48,"tag":171,"props":3270,"children":3271},{"__ignoreMap":728},[3272,3280],{"type":48,"tag":734,"props":3273,"children":3274},{"class":736,"line":737},[3275],{"type":48,"tag":734,"props":3276,"children":3277},{},[3278],{"type":54,"value":3279},"constexpr int BLOCK_SIZE = 256;\n",{"type":48,"tag":734,"props":3281,"children":3282},{"class":736,"line":746},[3283],{"type":48,"tag":734,"props":3284,"children":3285},{},[3286],{"type":54,"value":3287},"int num_blocks = (total_elements + BLOCK_SIZE - 1) \u002F BLOCK_SIZE;\n",{"type":48,"tag":57,"props":3289,"children":3290},{},[3291],{"type":54,"value":3292},"For reduction ops (LayerNorm, RMSNorm) with vectorization:",{"type":48,"tag":723,"props":3294,"children":3296},{"className":2952,"code":3295,"language":2954,"meta":728,"style":728},"\u002F\u002F Divide by 2 for bf16\u002Ffp16 vectorized access\nint threads = min(hidden_size \u002F 2, MAX_THREADS);\nthreads = max(threads, WARP_SIZE);\nthreads = (threads + 32 - 1) \u002F 32 * 32;  \u002F\u002F Round to warp boundary\n",[3297],{"type":48,"tag":171,"props":3298,"children":3299},{"__ignoreMap":728},[3300,3308,3316,3324],{"type":48,"tag":734,"props":3301,"children":3302},{"class":736,"line":737},[3303],{"type":48,"tag":734,"props":3304,"children":3305},{},[3306],{"type":54,"value":3307},"\u002F\u002F Divide by 2 for bf16\u002Ffp16 vectorized access\n",{"type":48,"tag":734,"props":3309,"children":3310},{"class":736,"line":746},[3311],{"type":48,"tag":734,"props":3312,"children":3313},{},[3314],{"type":54,"value":3315},"int threads = min(hidden_size \u002F 2, MAX_THREADS);\n",{"type":48,"tag":734,"props":3317,"children":3318},{"class":736,"line":755},[3319],{"type":48,"tag":734,"props":3320,"children":3321},{},[3322],{"type":54,"value":3323},"threads = max(threads, WARP_SIZE);\n",{"type":48,"tag":734,"props":3325,"children":3326},{"class":736,"line":764},[3327],{"type":48,"tag":734,"props":3328,"children":3329},{},[3330],{"type":54,"value":3331},"threads = (threads + 32 - 1) \u002F 32 * 32;  \u002F\u002F Round to warp boundary\n",{"type":48,"tag":78,"props":3333,"children":3335},{"id":3334},"supported-data-types",[3336],{"type":54,"value":3337},"Supported Data Types",{"type":48,"tag":57,"props":3339,"children":3340},{},[3341],{"type":54,"value":3342},"All kernels support three precision modes:",{"type":48,"tag":1625,"props":3344,"children":3345},{},[3346,3357,3368],{"type":48,"tag":961,"props":3347,"children":3348},{},[3349,3355],{"type":48,"tag":171,"props":3350,"children":3352},{"className":3351},[],[3353],{"type":54,"value":3354},"__half",{"type":54,"value":3356}," (FP16) - Default for inference",{"type":48,"tag":961,"props":3358,"children":3359},{},[3360,3366],{"type":48,"tag":171,"props":3361,"children":3363},{"className":3362},[],[3364],{"type":54,"value":3365},"__nv_bfloat16",{"type":54,"value":3367}," (BF16) - Preferred for training",{"type":48,"tag":961,"props":3369,"children":3370},{},[3371,3377],{"type":48,"tag":171,"props":3372,"children":3374},{"className":3373},[],[3375],{"type":54,"value":3376},"float",{"type":54,"value":3378}," (FP32) - Reference\u002Fdebugging",{"type":48,"tag":78,"props":3380,"children":3382},{"id":3381},"building-kernels",[3383],{"type":54,"value":3384},"Building Kernels",{"type":48,"tag":123,"props":3386,"children":3388},{"id":3387},"scaffold-a-new-kernel-project",[3389],{"type":54,"value":3390},"Scaffold a new kernel project",{"type":48,"tag":57,"props":3392,"children":3393},{},[3394,3396,3402],{"type":54,"value":3395},"Start new kernels with ",{"type":48,"tag":171,"props":3397,"children":3399},{"className":3398},[],[3400],{"type":54,"value":3401},"kernel-builder init",{"type":54,"value":3403}," instead of creating files by hand — it generates the compliant layout in one shot:",{"type":48,"tag":723,"props":3405,"children":3407},{"className":1119,"code":3406,"language":1121,"meta":728,"style":728},"kernel-builder init --name my-username\u002Fmy-kernel\n",[3408],{"type":48,"tag":171,"props":3409,"children":3410},{"__ignoreMap":728},[3411],{"type":48,"tag":734,"props":3412,"children":3413},{"class":736,"line":737},[3414,3418,3423,3428],{"type":48,"tag":734,"props":3415,"children":3416},{"style":1140},[3417],{"type":54,"value":96},{"type":48,"tag":734,"props":3419,"children":3420},{"style":1145},[3421],{"type":54,"value":3422}," init",{"type":48,"tag":734,"props":3424,"children":3425},{"style":1145},[3426],{"type":54,"value":3427}," --name",{"type":48,"tag":734,"props":3429,"children":3430},{"style":1145},[3431],{"type":54,"value":3432}," my-username\u002Fmy-kernel\n",{"type":48,"tag":57,"props":3434,"children":3435},{},[3436,3438,3443,3445,3451,3453,3459,3460,3465,3467,3473,3475,3481,3483,3489,3490,3496,3497,3503,3505,3511,3513,3519,3521,3526],{"type":54,"value":3437},"This creates ",{"type":48,"tag":171,"props":3439,"children":3441},{"className":3440},[],[3442],{"type":54,"value":328},{"type":54,"value":3444}," (valid dash-separated name, license, ",{"type":48,"tag":171,"props":3446,"children":3448},{"className":3447},[],[3449],{"type":54,"value":3450},"[general.hub] repo-id",{"type":54,"value":3452}," already wired), ",{"type":48,"tag":171,"props":3454,"children":3456},{"className":3455},[],[3457],{"type":54,"value":3458},"flake.nix",{"type":54,"value":178},{"type":48,"tag":171,"props":3461,"children":3463},{"className":3462},[],[3464],{"type":54,"value":565},{"type":54,"value":3466}," with compilable ",{"type":48,"tag":171,"props":3468,"children":3470},{"className":3469},[],[3471],{"type":54,"value":3472},"torch_binding.{h,cpp}",{"type":54,"value":3474}," and the Python package, a ",{"type":48,"tag":171,"props":3476,"children":3478},{"className":3477},[],[3479],{"type":54,"value":3480},"\u003Cname>_cuda\u002F",{"type":54,"value":3482}," kernel source dir, ",{"type":48,"tag":171,"props":3484,"children":3486},{"className":3485},[],[3487],{"type":54,"value":3488},"tests\u002F",{"type":54,"value":178},{"type":48,"tag":171,"props":3491,"children":3493},{"className":3492},[],[3494],{"type":54,"value":3495},"benchmarks\u002F",{"type":54,"value":178},{"type":48,"tag":171,"props":3498,"children":3500},{"className":3499},[],[3501],{"type":54,"value":3502},"example.py",{"type":54,"value":3504},", and ",{"type":48,"tag":171,"props":3506,"children":3508},{"className":3507},[],[3509],{"type":54,"value":3510},"CARD.md",{"type":54,"value":3512}," — and it initializes a git repository (required for builds). Then replace the stub kernel with your own sources and update the ",{"type":48,"tag":171,"props":3514,"children":3516},{"className":3515},[],[3517],{"type":54,"value":3518},"src",{"type":54,"value":3520}," lists in ",{"type":48,"tag":171,"props":3522,"children":3524},{"className":3523},[],[3525],{"type":54,"value":328},{"type":54,"value":1009},{"type":48,"tag":123,"props":3528,"children":3530},{"id":3529},"with-nix-recommended",[3531],{"type":54,"value":3532},"With Nix (Recommended)",{"type":48,"tag":723,"props":3534,"children":3536},{"className":1119,"code":3535,"language":1121,"meta":728,"style":728},"nix run .#build-and-copy --max-jobs 2 --cores 8 -L\n",[3537],{"type":48,"tag":171,"props":3538,"children":3539},{"__ignoreMap":728},[3540],{"type":48,"tag":734,"props":3541,"children":3542},{"class":736,"line":737},[3543,3548,3553,3558,3563,3568,3573,3578],{"type":48,"tag":734,"props":3544,"children":3545},{"style":1140},[3546],{"type":54,"value":3547},"nix",{"type":48,"tag":734,"props":3549,"children":3550},{"style":1145},[3551],{"type":54,"value":3552}," run",{"type":48,"tag":734,"props":3554,"children":3555},{"style":1145},[3556],{"type":54,"value":3557}," .#build-and-copy",{"type":48,"tag":734,"props":3559,"children":3560},{"style":1145},[3561],{"type":54,"value":3562}," --max-jobs",{"type":48,"tag":734,"props":3564,"children":3565},{"style":1842},[3566],{"type":54,"value":3567}," 2",{"type":48,"tag":734,"props":3569,"children":3570},{"style":1145},[3571],{"type":54,"value":3572}," --cores",{"type":48,"tag":734,"props":3574,"children":3575},{"style":1842},[3576],{"type":54,"value":3577}," 8",{"type":48,"tag":734,"props":3579,"children":3580},{"style":1145},[3581],{"type":54,"value":3582}," -L\n",{"type":48,"tag":123,"props":3584,"children":3586},{"id":3585},"build-and-publish-to-the-hub-in-one-go",[3587],{"type":54,"value":3588},"Build and publish to the Hub in one go",{"type":48,"tag":723,"props":3590,"children":3592},{"className":1119,"code":3591,"language":1121,"meta":728,"style":728},"kernel-builder build-and-upload\n",[3593],{"type":48,"tag":171,"props":3594,"children":3595},{"__ignoreMap":728},[3596],{"type":48,"tag":734,"props":3597,"children":3598},{"class":736,"line":737},[3599,3603],{"type":48,"tag":734,"props":3600,"children":3601},{"style":1140},[3602],{"type":54,"value":96},{"type":48,"tag":734,"props":3604,"children":3605},{"style":1145},[3606],{"type":54,"value":3607}," build-and-upload\n",{"type":48,"tag":57,"props":3609,"children":3610},{},[3611,3613,3619,3621,3627,3628,3634,3635,3640,3641,3646,3648,3659,3661,3668],{"type":54,"value":3612},"The target repo is set by ",{"type":48,"tag":171,"props":3614,"children":3616},{"className":3615},[],[3617],{"type":54,"value":3618},"repo-id",{"type":54,"value":3620}," under ",{"type":48,"tag":171,"props":3622,"children":3624},{"className":3623},[],[3625],{"type":54,"value":3626},"[general.hub]",{"type":54,"value":69},{"type":48,"tag":171,"props":3629,"children":3631},{"className":3630},[],[3632],{"type":54,"value":3633},"version",{"type":54,"value":3620},{"type":48,"tag":171,"props":3636,"children":3638},{"className":3637},[],[3639],{"type":54,"value":1026},{"type":54,"value":224},{"type":48,"tag":171,"props":3642,"children":3644},{"className":3643},[],[3645],{"type":54,"value":328},{"type":54,"value":3647},". Uploads go to a ",{"type":48,"tag":63,"props":3649,"children":3650},{},[3651,3657],{"type":48,"tag":171,"props":3652,"children":3654},{"className":3653},[],[3655],{"type":54,"value":3656},"kernel",{"type":54,"value":3658},"-type",{"type":54,"value":3660}," Hub repository (not a model repo); the owning user\u002Forg needs kernel-creation access (\"Request Kernels Creation\" at ",{"type":48,"tag":90,"props":3662,"children":3665},{"href":3663,"rel":3664},"https:\u002F\u002Fhuggingface.co\u002Fsettings\u002Faccount",[93],[3666],{"type":54,"value":3667},"huggingface.co\u002Fsettings\u002Faccount",{"type":54,"value":3669},").",{"type":48,"tag":123,"props":3671,"children":3673},{"id":3672},"local-build-for-development",[3674],{"type":54,"value":3675},"Local build for development",{"type":48,"tag":57,"props":3677,"children":3678},{},[3679,3681,3686,3688,3693,3695,3701,3703,3709],{"type":54,"value":3680},"Never hand-write a ",{"type":48,"tag":171,"props":3682,"children":3684},{"className":3683},[],[3685],{"type":54,"value":270},{"type":54,"value":3687}," (it leads to ",{"type":48,"tag":171,"props":3689,"children":3691},{"className":3690},[],[3692],{"type":54,"value":286},{"type":54,"value":3694},"\u002Fpybind11, which cannot build under ABI3). Let kernel-builder generate the project files, then build with ",{"type":48,"tag":171,"props":3696,"children":3698},{"className":3697},[],[3699],{"type":54,"value":3700},"setup.py build_kernel",{"type":54,"value":3702}," (no ",{"type":48,"tag":171,"props":3704,"children":3706},{"className":3705},[],[3707],{"type":54,"value":3708},"pip install",{"type":54,"value":3710},"\u002Feditable install needed):",{"type":48,"tag":723,"props":3712,"children":3714},{"className":1119,"code":3713,"language":1121,"meta":728,"style":728},"kernel-builder create-pyproject -f\npython setup.py build_kernel\n",[3715],{"type":48,"tag":171,"props":3716,"children":3717},{"__ignoreMap":728},[3718,3735],{"type":48,"tag":734,"props":3719,"children":3720},{"class":736,"line":737},[3721,3725,3730],{"type":48,"tag":734,"props":3722,"children":3723},{"style":1140},[3724],{"type":54,"value":96},{"type":48,"tag":734,"props":3726,"children":3727},{"style":1145},[3728],{"type":54,"value":3729}," create-pyproject",{"type":48,"tag":734,"props":3731,"children":3732},{"style":1145},[3733],{"type":54,"value":3734}," -f\n",{"type":48,"tag":734,"props":3736,"children":3737},{"class":736,"line":746},[3738,3742,3747],{"type":48,"tag":734,"props":3739,"children":3740},{"style":1140},[3741],{"type":54,"value":23},{"type":48,"tag":734,"props":3743,"children":3744},{"style":1145},[3745],{"type":54,"value":3746}," setup.py",{"type":48,"tag":734,"props":3748,"children":3749},{"style":1145},[3750],{"type":54,"value":3751}," build_kernel\n",{"type":48,"tag":57,"props":3753,"children":3754},{},[3755,3757,3763,3765,3771,3773,3779,3781,3787,3788,3794,3796,3802],{"type":54,"value":3756},"This builds the kernel and puts the output in ",{"type":48,"tag":171,"props":3758,"children":3760},{"className":3759},[],[3761],{"type":54,"value":3762},"build",{"type":54,"value":3764},", which can be loaded directly with ",{"type":48,"tag":171,"props":3766,"children":3768},{"className":3767},[],[3769],{"type":54,"value":3770},"kernels.get_local_kernel(Path(\"build\"))",{"type":54,"value":3772},". Inside ",{"type":48,"tag":171,"props":3774,"children":3776},{"className":3775},[],[3777],{"type":54,"value":3778},"kernel-builder devshell",{"type":54,"value":3780},"\u002F",{"type":48,"tag":171,"props":3782,"children":3784},{"className":3783},[],[3785],{"type":54,"value":3786},"testshell",{"type":54,"value":178},{"type":48,"tag":171,"props":3789,"children":3791},{"className":3790},[],[3792],{"type":54,"value":3793},"LOCAL_KERNELS",{"type":54,"value":3795}," is set automatically so ",{"type":48,"tag":171,"props":3797,"children":3799},{"className":3798},[],[3800],{"type":54,"value":3801},"get_kernel(\"\u003Crepo-id>\")",{"type":54,"value":3803}," resolves to this local build.",{"type":48,"tag":123,"props":3805,"children":3807},{"id":3806},"buildtoml-configuration",[3808],{"type":54,"value":3809},"build.toml Configuration",{"type":48,"tag":723,"props":3811,"children":3815},{"className":3812,"code":3813,"language":3814,"meta":728,"style":728},"language-toml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[general]\n# Name MUST be dash-separated lowercase (my-kernel), never underscores —\n# `kernel-builder check-config` rejects underscores. The Python package\n# lives at torch-ext\u002F\u003Cname with dashes replaced by underscores>.\nname = \"ltx-kernels\"\nbackends = [\"cuda\"]\nversion = 1\nlicense = \"Apache-2.0\"   # required field\n\n[general.hub]\n# Hub repo for `kernel-builder build-and-upload`; with `version` this\n# selects the version branch (e.g. v1).\nrepo-id = \"my-username\u002Fltx-kernels\"\n\n[torch]\nsrc = [\n  \"torch-ext\u002Ftorch_binding.cpp\",\n  \"torch-ext\u002Ftorch_binding.h\"\n]\n\n[kernel.your_kernel]\nbackend = \"cuda\"\nsrc = [\"kernel_src\u002Fyour_kernel.cu\"]\ndepends = [\"torch\"]\n# Only constrain cuda-capabilities when the kernel truly requires it —\n# do not over-specify.\n","toml",[3816],{"type":48,"tag":171,"props":3817,"children":3818},{"__ignoreMap":728},[3819,3827,3835,3843,3851,3859,3867,3875,3883,3890,3898,3907,3916,3925,3933,3942,3951,3960,3969,3978,3986,3995,4004,4013,4022,4031],{"type":48,"tag":734,"props":3820,"children":3821},{"class":736,"line":737},[3822],{"type":48,"tag":734,"props":3823,"children":3824},{},[3825],{"type":54,"value":3826},"[general]\n",{"type":48,"tag":734,"props":3828,"children":3829},{"class":736,"line":746},[3830],{"type":48,"tag":734,"props":3831,"children":3832},{},[3833],{"type":54,"value":3834},"# Name MUST be dash-separated lowercase (my-kernel), never underscores —\n",{"type":48,"tag":734,"props":3836,"children":3837},{"class":736,"line":755},[3838],{"type":48,"tag":734,"props":3839,"children":3840},{},[3841],{"type":54,"value":3842},"# `kernel-builder check-config` rejects underscores. The Python package\n",{"type":48,"tag":734,"props":3844,"children":3845},{"class":736,"line":764},[3846],{"type":48,"tag":734,"props":3847,"children":3848},{},[3849],{"type":54,"value":3850},"# lives at torch-ext\u002F\u003Cname with dashes replaced by underscores>.\n",{"type":48,"tag":734,"props":3852,"children":3853},{"class":736,"line":823},[3854],{"type":48,"tag":734,"props":3855,"children":3856},{},[3857],{"type":54,"value":3858},"name = \"ltx-kernels\"\n",{"type":48,"tag":734,"props":3860,"children":3861},{"class":736,"line":832},[3862],{"type":48,"tag":734,"props":3863,"children":3864},{},[3865],{"type":54,"value":3866},"backends = [\"cuda\"]\n",{"type":48,"tag":734,"props":3868,"children":3869},{"class":736,"line":841},[3870],{"type":48,"tag":734,"props":3871,"children":3872},{},[3873],{"type":54,"value":3874},"version = 1\n",{"type":48,"tag":734,"props":3876,"children":3877},{"class":736,"line":850},[3878],{"type":48,"tag":734,"props":3879,"children":3880},{},[3881],{"type":54,"value":3882},"license = \"Apache-2.0\"   # required field\n",{"type":48,"tag":734,"props":3884,"children":3885},{"class":736,"line":859},[3886],{"type":48,"tag":734,"props":3887,"children":3888},{"emptyLinePlaceholder":41},[3889],{"type":54,"value":761},{"type":48,"tag":734,"props":3891,"children":3892},{"class":736,"line":867},[3893],{"type":48,"tag":734,"props":3894,"children":3895},{},[3896],{"type":54,"value":3897},"[general.hub]\n",{"type":48,"tag":734,"props":3899,"children":3901},{"class":736,"line":3900},11,[3902],{"type":48,"tag":734,"props":3903,"children":3904},{},[3905],{"type":54,"value":3906},"# Hub repo for `kernel-builder build-and-upload`; with `version` this\n",{"type":48,"tag":734,"props":3908,"children":3910},{"class":736,"line":3909},12,[3911],{"type":48,"tag":734,"props":3912,"children":3913},{},[3914],{"type":54,"value":3915},"# selects the version branch (e.g. v1).\n",{"type":48,"tag":734,"props":3917,"children":3919},{"class":736,"line":3918},13,[3920],{"type":48,"tag":734,"props":3921,"children":3922},{},[3923],{"type":54,"value":3924},"repo-id = \"my-username\u002Fltx-kernels\"\n",{"type":48,"tag":734,"props":3926,"children":3928},{"class":736,"line":3927},14,[3929],{"type":48,"tag":734,"props":3930,"children":3931},{"emptyLinePlaceholder":41},[3932],{"type":54,"value":761},{"type":48,"tag":734,"props":3934,"children":3936},{"class":736,"line":3935},15,[3937],{"type":48,"tag":734,"props":3938,"children":3939},{},[3940],{"type":54,"value":3941},"[torch]\n",{"type":48,"tag":734,"props":3943,"children":3945},{"class":736,"line":3944},16,[3946],{"type":48,"tag":734,"props":3947,"children":3948},{},[3949],{"type":54,"value":3950},"src = [\n",{"type":48,"tag":734,"props":3952,"children":3954},{"class":736,"line":3953},17,[3955],{"type":48,"tag":734,"props":3956,"children":3957},{},[3958],{"type":54,"value":3959},"  \"torch-ext\u002Ftorch_binding.cpp\",\n",{"type":48,"tag":734,"props":3961,"children":3963},{"class":736,"line":3962},18,[3964],{"type":48,"tag":734,"props":3965,"children":3966},{},[3967],{"type":54,"value":3968},"  \"torch-ext\u002Ftorch_binding.h\"\n",{"type":48,"tag":734,"props":3970,"children":3972},{"class":736,"line":3971},19,[3973],{"type":48,"tag":734,"props":3974,"children":3975},{},[3976],{"type":54,"value":3977},"]\n",{"type":48,"tag":734,"props":3979,"children":3981},{"class":736,"line":3980},20,[3982],{"type":48,"tag":734,"props":3983,"children":3984},{"emptyLinePlaceholder":41},[3985],{"type":54,"value":761},{"type":48,"tag":734,"props":3987,"children":3989},{"class":736,"line":3988},21,[3990],{"type":48,"tag":734,"props":3991,"children":3992},{},[3993],{"type":54,"value":3994},"[kernel.your_kernel]\n",{"type":48,"tag":734,"props":3996,"children":3998},{"class":736,"line":3997},22,[3999],{"type":48,"tag":734,"props":4000,"children":4001},{},[4002],{"type":54,"value":4003},"backend = \"cuda\"\n",{"type":48,"tag":734,"props":4005,"children":4007},{"class":736,"line":4006},23,[4008],{"type":48,"tag":734,"props":4009,"children":4010},{},[4011],{"type":54,"value":4012},"src = [\"kernel_src\u002Fyour_kernel.cu\"]\n",{"type":48,"tag":734,"props":4014,"children":4016},{"class":736,"line":4015},24,[4017],{"type":48,"tag":734,"props":4018,"children":4019},{},[4020],{"type":54,"value":4021},"depends = [\"torch\"]\n",{"type":48,"tag":734,"props":4023,"children":4025},{"class":736,"line":4024},25,[4026],{"type":48,"tag":734,"props":4027,"children":4028},{},[4029],{"type":54,"value":4030},"# Only constrain cuda-capabilities when the kernel truly requires it —\n",{"type":48,"tag":734,"props":4032,"children":4034},{"class":736,"line":4033},26,[4035],{"type":48,"tag":734,"props":4036,"children":4037},{},[4038],{"type":54,"value":4039},"# do not over-specify.\n",{"type":48,"tag":57,"props":4041,"children":4042},{},[4043,4045,4050,4051,4057],{"type":54,"value":4044},"The kernel directory ",{"type":48,"tag":63,"props":4046,"children":4047},{},[4048],{"type":54,"value":4049},"must be a git repository with files committed",{"type":54,"value":288},{"type":48,"tag":171,"props":4052,"children":4054},{"className":4053},[],[4055],{"type":54,"value":4056},"git init && git add -A && git commit",{"type":54,"value":4058},") — Nix refuses to build non-git kernels (\"Kernel is not in a git repository\").",{"type":48,"tag":78,"props":4060,"children":4062},{"id":4061},"library-integration",[4063],{"type":54,"value":4064},"Library Integration",{"type":48,"tag":123,"props":4066,"children":4068},{"id":4067},"huggingface-kernels-hub-get_kernel",[4069],{"type":54,"value":4070},"HuggingFace Kernels Hub (get_kernel)",{"type":48,"tag":2900,"props":4072,"children":4073},{},[4074],{"type":48,"tag":57,"props":4075,"children":4076},{},[4077],{"type":48,"tag":63,"props":4078,"children":4079},{},[4080,4082,4088],{"type":54,"value":4081},"See ",{"type":48,"tag":90,"props":4083,"children":4085},{"href":4084},"references\u002Fhuggingface-kernels-integration.md",[4086],{"type":54,"value":4087},"huggingface-kernels-integration.md",{"type":54,"value":4089}," for the complete guide.",{"type":48,"tag":57,"props":4091,"children":4092},{},[4093],{"type":54,"value":4094},"Load pre-compiled, optimized kernels directly from HuggingFace Hub without local compilation:",{"type":48,"tag":723,"props":4096,"children":4098},{"className":889,"code":4097,"language":23,"meta":728,"style":728},"from kernels import get_kernel, has_kernel\n\n# Check availability and load — Hub loads REQUIRE version= (or revision=);\n# a bare get_kernel(repo_id) raises ValueError.\nif has_kernel(\"kernels-community\u002Factivation\", version=1):\n    activation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n\n    # Use the kernel\n    x = torch.randn((4, 4), dtype=torch.float16, device=\"cuda\")\n    y = torch.empty_like(x)\n    activation.gelu_fast(y, x)\n",[4099],{"type":48,"tag":171,"props":4100,"children":4101},{"__ignoreMap":728},[4102,4110,4117,4125,4133,4141,4149,4156,4164,4172,4180],{"type":48,"tag":734,"props":4103,"children":4104},{"class":736,"line":737},[4105],{"type":48,"tag":734,"props":4106,"children":4107},{},[4108],{"type":54,"value":4109},"from kernels import get_kernel, has_kernel\n",{"type":48,"tag":734,"props":4111,"children":4112},{"class":736,"line":746},[4113],{"type":48,"tag":734,"props":4114,"children":4115},{"emptyLinePlaceholder":41},[4116],{"type":54,"value":761},{"type":48,"tag":734,"props":4118,"children":4119},{"class":736,"line":755},[4120],{"type":48,"tag":734,"props":4121,"children":4122},{},[4123],{"type":54,"value":4124},"# Check availability and load — Hub loads REQUIRE version= (or revision=);\n",{"type":48,"tag":734,"props":4126,"children":4127},{"class":736,"line":764},[4128],{"type":48,"tag":734,"props":4129,"children":4130},{},[4131],{"type":54,"value":4132},"# a bare get_kernel(repo_id) raises ValueError.\n",{"type":48,"tag":734,"props":4134,"children":4135},{"class":736,"line":823},[4136],{"type":48,"tag":734,"props":4137,"children":4138},{},[4139],{"type":54,"value":4140},"if has_kernel(\"kernels-community\u002Factivation\", version=1):\n",{"type":48,"tag":734,"props":4142,"children":4143},{"class":736,"line":832},[4144],{"type":48,"tag":734,"props":4145,"children":4146},{},[4147],{"type":54,"value":4148},"    activation = get_kernel(\"kernels-community\u002Factivation\", version=1)\n",{"type":48,"tag":734,"props":4150,"children":4151},{"class":736,"line":841},[4152],{"type":48,"tag":734,"props":4153,"children":4154},{"emptyLinePlaceholder":41},[4155],{"type":54,"value":761},{"type":48,"tag":734,"props":4157,"children":4158},{"class":736,"line":850},[4159],{"type":48,"tag":734,"props":4160,"children":4161},{},[4162],{"type":54,"value":4163},"    # Use the kernel\n",{"type":48,"tag":734,"props":4165,"children":4166},{"class":736,"line":859},[4167],{"type":48,"tag":734,"props":4168,"children":4169},{},[4170],{"type":54,"value":4171},"    x = torch.randn((4, 4), dtype=torch.float16, device=\"cuda\")\n",{"type":48,"tag":734,"props":4173,"children":4174},{"class":736,"line":867},[4175],{"type":48,"tag":734,"props":4176,"children":4177},{},[4178],{"type":54,"value":4179},"    y = torch.empty_like(x)\n",{"type":48,"tag":734,"props":4181,"children":4182},{"class":736,"line":3900},[4183],{"type":48,"tag":734,"props":4184,"children":4185},{},[4186],{"type":54,"value":4187},"    activation.gelu_fast(y, x)\n",{"type":48,"tag":57,"props":4189,"children":4190},{},[4191],{"type":48,"tag":63,"props":4192,"children":4193},{},[4194],{"type":54,"value":4195},"Key functions:",{"type":48,"tag":1625,"props":4197,"children":4198},{},[4199,4231,4242],{"type":48,"tag":961,"props":4200,"children":4201},{},[4202,4208,4210,4216,4218,4224,4226],{"type":48,"tag":171,"props":4203,"children":4205},{"className":4204},[],[4206],{"type":54,"value":4207},"get_kernel(repo_id, version=N)",{"type":54,"value":4209}," - Download and load kernel from Hub; ",{"type":48,"tag":171,"props":4211,"children":4213},{"className":4212},[],[4214],{"type":54,"value":4215},"version=",{"type":54,"value":4217}," (major version) or ",{"type":48,"tag":171,"props":4219,"children":4221},{"className":4220},[],[4222],{"type":54,"value":4223},"revision=",{"type":54,"value":4225}," (branch\u002Ftag\u002Fcommit) is ",{"type":48,"tag":63,"props":4227,"children":4228},{},[4229],{"type":54,"value":4230},"required",{"type":48,"tag":961,"props":4232,"children":4233},{},[4234,4240],{"type":48,"tag":171,"props":4235,"children":4237},{"className":4236},[],[4238],{"type":54,"value":4239},"has_kernel(repo_id, version=N)",{"type":54,"value":4241}," - Check if compatible build exists",{"type":48,"tag":961,"props":4243,"children":4244},{},[4245,4251,4253,4259,4260,4266],{"type":48,"tag":171,"props":4246,"children":4248},{"className":4247},[],[4249],{"type":54,"value":4250},"get_local_kernel(Path(\"path\u002Fto\u002Fkernel-project\"))",{"type":54,"value":4252}," - Load a local build (looks in ",{"type":48,"tag":171,"props":4254,"children":4256},{"className":4255},[],[4257],{"type":54,"value":4258},"\u003Cpath>",{"type":54,"value":69},{"type":48,"tag":171,"props":4261,"children":4263},{"className":4262},[],[4264],{"type":54,"value":4265},"\u003Cpath>\u002Fbuild",{"type":54,"value":4267},") — use during development",{"type":48,"tag":57,"props":4269,"children":4270},{},[4271,4284,4286,4292,4294,4300],{"type":48,"tag":63,"props":4272,"children":4273},{},[4274,4276,4282],{"type":54,"value":4275},"Testing local builds through the ",{"type":48,"tag":171,"props":4277,"children":4279},{"className":4278},[],[4280],{"type":54,"value":4281},"get_kernel()",{"type":54,"value":4283}," code path:",{"type":54,"value":4285}," set ",{"type":48,"tag":171,"props":4287,"children":4289},{"className":4288},[],[4290],{"type":54,"value":4291},"LOCAL_KERNELS=\"org\u002Fname=\u002Fpath\u002Fto\u002Fkernel-project\"",{"type":54,"value":4293}," and call ",{"type":48,"tag":171,"props":4295,"children":4297},{"className":4296},[],[4298],{"type":54,"value":4299},"get_kernel(\"org\u002Fname\")",{"type":54,"value":4301}," unchanged — the override short-circuits the Hub entirely (no download, no version needed), so integration code can be tested verbatim against a local build.",{"type":48,"tag":57,"props":4303,"children":4304},{},[4305],{"type":48,"tag":63,"props":4306,"children":4307},{},[4308],{"type":54,"value":4309},"Popular community kernels:",{"type":48,"tag":1625,"props":4311,"children":4312},{},[4313,4324,4335],{"type":48,"tag":961,"props":4314,"children":4315},{},[4316,4322],{"type":48,"tag":171,"props":4317,"children":4319},{"className":4318},[],[4320],{"type":54,"value":4321},"kernels-community\u002Factivation",{"type":54,"value":4323}," - GELU, SiLU, etc.",{"type":48,"tag":961,"props":4325,"children":4326},{},[4327,4333],{"type":48,"tag":171,"props":4328,"children":4330},{"className":4329},[],[4331],{"type":54,"value":4332},"kernels-community\u002Fflash-attn",{"type":54,"value":4334}," - Flash Attention 2",{"type":48,"tag":961,"props":4336,"children":4337},{},[4338,4344],{"type":48,"tag":171,"props":4339,"children":4341},{"className":4340},[],[4342],{"type":54,"value":4343},"kernels-community\u002Ftriton-layer-norm",{"type":54,"value":4345}," - LayerNorm, RMSNorm",{"type":48,"tag":123,"props":4347,"children":4349},{"id":4348},"diffusers-integration-videoimage-generation",[4350],{"type":54,"value":4351},"Diffusers Integration (Video\u002FImage Generation)",{"type":48,"tag":2900,"props":4353,"children":4354},{},[4355],{"type":48,"tag":57,"props":4356,"children":4357},{},[4358],{"type":48,"tag":63,"props":4359,"children":4360},{},[4361,4362,4368],{"type":54,"value":4081},{"type":48,"tag":90,"props":4363,"children":4365},{"href":4364},"references\u002Fdiffusers-integration.md",[4366],{"type":54,"value":4367},"diffusers-integration.md",{"type":54,"value":4089},{"type":48,"tag":123,"props":4370,"children":4372},{"id":4371},"transformers-integration-llms",[4373],{"type":54,"value":4374},"Transformers Integration (LLMs)",{"type":48,"tag":2900,"props":4376,"children":4377},{},[4378],{"type":48,"tag":57,"props":4379,"children":4380},{},[4381],{"type":48,"tag":63,"props":4382,"children":4383},{},[4384,4385,4391],{"type":54,"value":4081},{"type":48,"tag":90,"props":4386,"children":4388},{"href":4387},"references\u002Ftransformers-integration.md",[4389],{"type":54,"value":4390},"transformers-integration.md",{"type":54,"value":4089},{"type":48,"tag":57,"props":4393,"children":4394},{},[4395],{"type":48,"tag":63,"props":4396,"children":4397},{},[4398],{"type":54,"value":4399},"Key differences from diffusers:",{"type":48,"tag":1625,"props":4401,"children":4402},{},[4403,4422,4435,4456],{"type":48,"tag":961,"props":4404,"children":4405},{},[4406,4408,4413,4415,4421],{"type":54,"value":4407},"Transformers RMSNorm ",{"type":48,"tag":63,"props":4409,"children":4410},{},[4411],{"type":54,"value":4412},"always",{"type":54,"value":4414}," has weights (no ",{"type":48,"tag":171,"props":4416,"children":4418},{"className":4417},[],[4419],{"type":54,"value":4420},"elementwise_affine=False",{"type":54,"value":317},{"type":48,"tag":961,"props":4423,"children":4424},{},[4425,4427,4433],{"type":54,"value":4426},"Use ",{"type":48,"tag":171,"props":4428,"children":4430},{"className":4429},[],[4431],{"type":54,"value":4432},"'RMSNorm' in class_name",{"type":54,"value":4434}," to match LlamaRMSNorm, MistralRMSNorm, etc.",{"type":48,"tag":961,"props":4436,"children":4437},{},[4438,4440,4446,4448,4454],{"type":54,"value":4439},"Check for ",{"type":48,"tag":171,"props":4441,"children":4443},{"className":4442},[],[4444],{"type":54,"value":4445},"variance_epsilon",{"type":54,"value":4447}," (LLaMA) or ",{"type":48,"tag":171,"props":4449,"children":4451},{"className":4450},[],[4452],{"type":54,"value":4453},"eps",{"type":54,"value":4455}," (others) for epsilon",{"type":48,"tag":961,"props":4457,"children":4458},{},[4459,4460,4466],{"type":54,"value":994},{"type":48,"tag":171,"props":4461,"children":4463},{"className":4462},[],[4464],{"type":54,"value":4465},"set_processor()",{"type":54,"value":4467}," pattern - use Flash Attention 2 instead",{"type":48,"tag":57,"props":4469,"children":4470},{},[4471],{"type":48,"tag":63,"props":4472,"children":4473},{},[4474],{"type":54,"value":4475},"Minimal transformers pattern:",{"type":48,"tag":723,"props":4477,"children":4479},{"className":889,"code":4478,"language":23,"meta":728,"style":728},"from transformers import AutoModelForCausalLM\nfrom ltx_kernels import rmsnorm\n\ndef patch_rmsnorm(model):\n    for name, module in model.named_modules():\n        if 'RMSNorm' in type(module).__name__:\n            eps = getattr(module, 'variance_epsilon', None) or getattr(module, 'eps', 1e-6)\n            def make_forward(mod, epsilon):\n                def forward(x):\n                    return rmsnorm(x, mod.weight, eps=epsilon)\n                return forward\n            module.forward = make_forward(module, eps)\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\", torch_dtype=torch.bfloat16)\npatch_rmsnorm(model)\n",[4480],{"type":48,"tag":171,"props":4481,"children":4482},{"__ignoreMap":728},[4483,4491,4499,4506,4514,4522,4530,4538,4546,4554,4562,4570,4578,4585,4593],{"type":48,"tag":734,"props":4484,"children":4485},{"class":736,"line":737},[4486],{"type":48,"tag":734,"props":4487,"children":4488},{},[4489],{"type":54,"value":4490},"from transformers import AutoModelForCausalLM\n",{"type":48,"tag":734,"props":4492,"children":4493},{"class":736,"line":746},[4494],{"type":48,"tag":734,"props":4495,"children":4496},{},[4497],{"type":54,"value":4498},"from ltx_kernels import rmsnorm\n",{"type":48,"tag":734,"props":4500,"children":4501},{"class":736,"line":755},[4502],{"type":48,"tag":734,"props":4503,"children":4504},{"emptyLinePlaceholder":41},[4505],{"type":54,"value":761},{"type":48,"tag":734,"props":4507,"children":4508},{"class":736,"line":764},[4509],{"type":48,"tag":734,"props":4510,"children":4511},{},[4512],{"type":54,"value":4513},"def patch_rmsnorm(model):\n",{"type":48,"tag":734,"props":4515,"children":4516},{"class":736,"line":823},[4517],{"type":48,"tag":734,"props":4518,"children":4519},{},[4520],{"type":54,"value":4521},"    for name, module in model.named_modules():\n",{"type":48,"tag":734,"props":4523,"children":4524},{"class":736,"line":832},[4525],{"type":48,"tag":734,"props":4526,"children":4527},{},[4528],{"type":54,"value":4529},"        if 'RMSNorm' in type(module).__name__:\n",{"type":48,"tag":734,"props":4531,"children":4532},{"class":736,"line":841},[4533],{"type":48,"tag":734,"props":4534,"children":4535},{},[4536],{"type":54,"value":4537},"            eps = getattr(module, 'variance_epsilon', None) or getattr(module, 'eps', 1e-6)\n",{"type":48,"tag":734,"props":4539,"children":4540},{"class":736,"line":850},[4541],{"type":48,"tag":734,"props":4542,"children":4543},{},[4544],{"type":54,"value":4545},"            def make_forward(mod, epsilon):\n",{"type":48,"tag":734,"props":4547,"children":4548},{"class":736,"line":859},[4549],{"type":48,"tag":734,"props":4550,"children":4551},{},[4552],{"type":54,"value":4553},"                def forward(x):\n",{"type":48,"tag":734,"props":4555,"children":4556},{"class":736,"line":867},[4557],{"type":48,"tag":734,"props":4558,"children":4559},{},[4560],{"type":54,"value":4561},"                    return rmsnorm(x, mod.weight, eps=epsilon)\n",{"type":48,"tag":734,"props":4563,"children":4564},{"class":736,"line":3900},[4565],{"type":48,"tag":734,"props":4566,"children":4567},{},[4568],{"type":54,"value":4569},"                return forward\n",{"type":48,"tag":734,"props":4571,"children":4572},{"class":736,"line":3909},[4573],{"type":48,"tag":734,"props":4574,"children":4575},{},[4576],{"type":54,"value":4577},"            module.forward = make_forward(module, eps)\n",{"type":48,"tag":734,"props":4579,"children":4580},{"class":736,"line":3918},[4581],{"type":48,"tag":734,"props":4582,"children":4583},{"emptyLinePlaceholder":41},[4584],{"type":54,"value":761},{"type":48,"tag":734,"props":4586,"children":4587},{"class":736,"line":3927},[4588],{"type":48,"tag":734,"props":4589,"children":4590},{},[4591],{"type":54,"value":4592},"model = AutoModelForCausalLM.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\", torch_dtype=torch.bfloat16)\n",{"type":48,"tag":734,"props":4594,"children":4595},{"class":736,"line":3935},[4596],{"type":48,"tag":734,"props":4597,"children":4598},{},[4599],{"type":54,"value":4600},"patch_rmsnorm(model)\n",{"type":48,"tag":123,"props":4602,"children":4604},{"id":4603},"diffusers-critical-pitfalls",[4605],{"type":54,"value":4606},"Diffusers Critical Pitfalls",{"type":48,"tag":4608,"props":4609,"children":4611},"h4",{"id":4610},"_1-rmsnorm-weight-may-be-none",[4612],{"type":54,"value":4613},"1. RMSNorm Weight May Be None",{"type":48,"tag":57,"props":4615,"children":4616},{},[4617,4619,4624],{"type":54,"value":4618},"LTX-Video uses ",{"type":48,"tag":171,"props":4620,"children":4622},{"className":4621},[],[4623],{"type":54,"value":4420},{"type":54,"value":4625}," for some RMSNorm modules:",{"type":48,"tag":723,"props":4627,"children":4629},{"className":889,"code":4628,"language":23,"meta":728,"style":728},"# Transformer blocks: NO WEIGHT\nself.norm1 = RMSNorm(dim, elementwise_affine=False)\n\n# Attention modules: HAS WEIGHT\nself.norm_q = torch.nn.RMSNorm(..., elementwise_affine=True)\n",[4630],{"type":48,"tag":171,"props":4631,"children":4632},{"__ignoreMap":728},[4633,4641,4649,4656,4664],{"type":48,"tag":734,"props":4634,"children":4635},{"class":736,"line":737},[4636],{"type":48,"tag":734,"props":4637,"children":4638},{},[4639],{"type":54,"value":4640},"# Transformer blocks: NO WEIGHT\n",{"type":48,"tag":734,"props":4642,"children":4643},{"class":736,"line":746},[4644],{"type":48,"tag":734,"props":4645,"children":4646},{},[4647],{"type":54,"value":4648},"self.norm1 = RMSNorm(dim, elementwise_affine=False)\n",{"type":48,"tag":734,"props":4650,"children":4651},{"class":736,"line":755},[4652],{"type":48,"tag":734,"props":4653,"children":4654},{"emptyLinePlaceholder":41},[4655],{"type":54,"value":761},{"type":48,"tag":734,"props":4657,"children":4658},{"class":736,"line":764},[4659],{"type":48,"tag":734,"props":4660,"children":4661},{},[4662],{"type":54,"value":4663},"# Attention modules: HAS WEIGHT\n",{"type":48,"tag":734,"props":4665,"children":4666},{"class":736,"line":823},[4667],{"type":48,"tag":734,"props":4668,"children":4669},{},[4670],{"type":54,"value":4671},"self.norm_q = torch.nn.RMSNorm(..., elementwise_affine=True)\n",{"type":48,"tag":57,"props":4673,"children":4674},{},[4675,4680],{"type":48,"tag":63,"props":4676,"children":4677},{},[4678],{"type":54,"value":4679},"Solution:",{"type":54,"value":4681}," Handle both cases:",{"type":48,"tag":723,"props":4683,"children":4685},{"className":889,"code":4684,"language":23,"meta":728,"style":728},"has_weight = hasattr(module, 'weight') and module.weight is not None\nif has_weight:\n    output = rmsnorm(x, module.weight, eps=eps)\nelse:\n    weight = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n    output = rmsnorm(x, weight, eps=eps)\n",[4686],{"type":48,"tag":171,"props":4687,"children":4688},{"__ignoreMap":728},[4689,4697,4705,4713,4721,4729],{"type":48,"tag":734,"props":4690,"children":4691},{"class":736,"line":737},[4692],{"type":48,"tag":734,"props":4693,"children":4694},{},[4695],{"type":54,"value":4696},"has_weight = hasattr(module, 'weight') and module.weight is not None\n",{"type":48,"tag":734,"props":4698,"children":4699},{"class":736,"line":746},[4700],{"type":48,"tag":734,"props":4701,"children":4702},{},[4703],{"type":54,"value":4704},"if has_weight:\n",{"type":48,"tag":734,"props":4706,"children":4707},{"class":736,"line":755},[4708],{"type":48,"tag":734,"props":4709,"children":4710},{},[4711],{"type":54,"value":4712},"    output = rmsnorm(x, module.weight, eps=eps)\n",{"type":48,"tag":734,"props":4714,"children":4715},{"class":736,"line":764},[4716],{"type":48,"tag":734,"props":4717,"children":4718},{},[4719],{"type":54,"value":4720},"else:\n",{"type":48,"tag":734,"props":4722,"children":4723},{"class":736,"line":823},[4724],{"type":48,"tag":734,"props":4725,"children":4726},{},[4727],{"type":54,"value":4728},"    weight = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n",{"type":48,"tag":734,"props":4730,"children":4731},{"class":736,"line":832},[4732],{"type":48,"tag":734,"props":4733,"children":4734},{},[4735],{"type":54,"value":4736},"    output = rmsnorm(x, weight, eps=eps)\n",{"type":48,"tag":4608,"props":4738,"children":4740},{"id":4739},"_2-diffusers-rmsnorm-torchnnrmsnorm",[4741],{"type":54,"value":4742},"2. Diffusers RMSNorm != torch.nn.RMSNorm",{"type":48,"tag":723,"props":4744,"children":4746},{"className":889,"code":4745,"language":23,"meta":728,"style":728},"# WRONG - misses diffusers RMSNorm\nif isinstance(module, torch.nn.RMSNorm):\n\n# CORRECT - catches all RMSNorm variants\nif type(module).__name__ == 'RMSNorm':\n",[4747],{"type":48,"tag":171,"props":4748,"children":4749},{"__ignoreMap":728},[4750,4758,4766,4773,4781],{"type":48,"tag":734,"props":4751,"children":4752},{"class":736,"line":737},[4753],{"type":48,"tag":734,"props":4754,"children":4755},{},[4756],{"type":54,"value":4757},"# WRONG - misses diffusers RMSNorm\n",{"type":48,"tag":734,"props":4759,"children":4760},{"class":736,"line":746},[4761],{"type":48,"tag":734,"props":4762,"children":4763},{},[4764],{"type":54,"value":4765},"if isinstance(module, torch.nn.RMSNorm):\n",{"type":48,"tag":734,"props":4767,"children":4768},{"class":736,"line":755},[4769],{"type":48,"tag":734,"props":4770,"children":4771},{"emptyLinePlaceholder":41},[4772],{"type":54,"value":761},{"type":48,"tag":734,"props":4774,"children":4775},{"class":736,"line":764},[4776],{"type":48,"tag":734,"props":4777,"children":4778},{},[4779],{"type":54,"value":4780},"# CORRECT - catches all RMSNorm variants\n",{"type":48,"tag":734,"props":4782,"children":4783},{"class":736,"line":823},[4784],{"type":48,"tag":734,"props":4785,"children":4786},{},[4787],{"type":54,"value":4788},"if type(module).__name__ == 'RMSNorm':\n",{"type":48,"tag":4608,"props":4790,"children":4792},{"id":4791},"_3-ltx-video-uses-gelu-not-geglu",[4793],{"type":54,"value":4794},"3. LTX-Video Uses GELU, Not GEGLU",{"type":48,"tag":57,"props":4796,"children":4797},{},[4798,4799,4805],{"type":54,"value":4618},{"type":48,"tag":171,"props":4800,"children":4802},{"className":4801},[],[4803],{"type":54,"value":4804},"activation_fn=\"gelu-approximate\"",{"type":54,"value":4806},". Don't patch GEGLU for LTX-Video.",{"type":48,"tag":4608,"props":4808,"children":4810},{"id":4809},"_4-inject-kernels-before-cpu-offloading",[4811],{"type":54,"value":4812},"4. Inject Kernels BEFORE CPU Offloading",{"type":48,"tag":723,"props":4814,"children":4816},{"className":889,"code":4815,"language":23,"meta":728,"style":728},"pipe = LTXPipeline.from_pretrained(...)\npipe.to(\"cuda\")\ninject_optimized_kernels(pipe)  # BEFORE offloading\npipe.enable_model_cpu_offload()  # Now safe\n",[4817],{"type":48,"tag":171,"props":4818,"children":4819},{"__ignoreMap":728},[4820,4828,4836,4844],{"type":48,"tag":734,"props":4821,"children":4822},{"class":736,"line":737},[4823],{"type":48,"tag":734,"props":4824,"children":4825},{},[4826],{"type":54,"value":4827},"pipe = LTXPipeline.from_pretrained(...)\n",{"type":48,"tag":734,"props":4829,"children":4830},{"class":736,"line":746},[4831],{"type":48,"tag":734,"props":4832,"children":4833},{},[4834],{"type":54,"value":4835},"pipe.to(\"cuda\")\n",{"type":48,"tag":734,"props":4837,"children":4838},{"class":736,"line":755},[4839],{"type":48,"tag":734,"props":4840,"children":4841},{},[4842],{"type":54,"value":4843},"inject_optimized_kernels(pipe)  # BEFORE offloading\n",{"type":48,"tag":734,"props":4845,"children":4846},{"class":736,"line":764},[4847],{"type":48,"tag":734,"props":4848,"children":4849},{},[4850],{"type":54,"value":4851},"pipe.enable_model_cpu_offload()  # Now safe\n",{"type":48,"tag":123,"props":4853,"children":4855},{"id":4854},"minimal-integration-pattern",[4856],{"type":54,"value":4857},"Minimal Integration Pattern",{"type":48,"tag":723,"props":4859,"children":4861},{"className":889,"code":4860,"language":23,"meta":728,"style":728},"from diffusers import LTXPipeline\nfrom ltx_kernels import rmsnorm\n\ndef patch_rmsnorm_modules(model):\n    \"\"\"Patch all RMSNorm modules to use custom kernel.\"\"\"\n    for name, module in model.named_modules():\n        if type(module).__name__ == 'RMSNorm':\n            eps = getattr(module, 'eps', 1e-6)\n            has_weight = hasattr(module, 'weight') and module.weight is not None\n\n            if has_weight:\n                def make_forward(mod, epsilon):\n                    def forward(x):\n                        return rmsnorm(x, mod.weight, eps=epsilon)\n                    return forward\n                module.forward = make_forward(module, eps)\n            else:\n                def make_forward(epsilon):\n                    def forward(x):\n                        w = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n                        return rmsnorm(x, w, eps=epsilon)\n                    return forward\n                module.forward = make_forward(eps)\n\n# Usage\npipe = LTXPipeline.from_pretrained(\"Lightricks\u002FLTX-Video\", torch_dtype=torch.bfloat16)\npipe.to(\"cuda\")\npatch_rmsnorm_modules(pipe.transformer)\npipe.enable_model_cpu_offload()\n",[4862],{"type":48,"tag":171,"props":4863,"children":4864},{"__ignoreMap":728},[4865,4873,4880,4887,4895,4903,4910,4918,4926,4934,4941,4949,4957,4965,4973,4981,4989,4997,5005,5012,5020,5028,5035,5043,5050,5058,5066,5074,5083],{"type":48,"tag":734,"props":4866,"children":4867},{"class":736,"line":737},[4868],{"type":48,"tag":734,"props":4869,"children":4870},{},[4871],{"type":54,"value":4872},"from diffusers import LTXPipeline\n",{"type":48,"tag":734,"props":4874,"children":4875},{"class":736,"line":746},[4876],{"type":48,"tag":734,"props":4877,"children":4878},{},[4879],{"type":54,"value":4498},{"type":48,"tag":734,"props":4881,"children":4882},{"class":736,"line":755},[4883],{"type":48,"tag":734,"props":4884,"children":4885},{"emptyLinePlaceholder":41},[4886],{"type":54,"value":761},{"type":48,"tag":734,"props":4888,"children":4889},{"class":736,"line":764},[4890],{"type":48,"tag":734,"props":4891,"children":4892},{},[4893],{"type":54,"value":4894},"def patch_rmsnorm_modules(model):\n",{"type":48,"tag":734,"props":4896,"children":4897},{"class":736,"line":823},[4898],{"type":48,"tag":734,"props":4899,"children":4900},{},[4901],{"type":54,"value":4902},"    \"\"\"Patch all RMSNorm modules to use custom kernel.\"\"\"\n",{"type":48,"tag":734,"props":4904,"children":4905},{"class":736,"line":832},[4906],{"type":48,"tag":734,"props":4907,"children":4908},{},[4909],{"type":54,"value":4521},{"type":48,"tag":734,"props":4911,"children":4912},{"class":736,"line":841},[4913],{"type":48,"tag":734,"props":4914,"children":4915},{},[4916],{"type":54,"value":4917},"        if type(module).__name__ == 'RMSNorm':\n",{"type":48,"tag":734,"props":4919,"children":4920},{"class":736,"line":850},[4921],{"type":48,"tag":734,"props":4922,"children":4923},{},[4924],{"type":54,"value":4925},"            eps = getattr(module, 'eps', 1e-6)\n",{"type":48,"tag":734,"props":4927,"children":4928},{"class":736,"line":859},[4929],{"type":48,"tag":734,"props":4930,"children":4931},{},[4932],{"type":54,"value":4933},"            has_weight = hasattr(module, 'weight') and module.weight is not None\n",{"type":48,"tag":734,"props":4935,"children":4936},{"class":736,"line":867},[4937],{"type":48,"tag":734,"props":4938,"children":4939},{"emptyLinePlaceholder":41},[4940],{"type":54,"value":761},{"type":48,"tag":734,"props":4942,"children":4943},{"class":736,"line":3900},[4944],{"type":48,"tag":734,"props":4945,"children":4946},{},[4947],{"type":54,"value":4948},"            if has_weight:\n",{"type":48,"tag":734,"props":4950,"children":4951},{"class":736,"line":3909},[4952],{"type":48,"tag":734,"props":4953,"children":4954},{},[4955],{"type":54,"value":4956},"                def make_forward(mod, epsilon):\n",{"type":48,"tag":734,"props":4958,"children":4959},{"class":736,"line":3918},[4960],{"type":48,"tag":734,"props":4961,"children":4962},{},[4963],{"type":54,"value":4964},"                    def forward(x):\n",{"type":48,"tag":734,"props":4966,"children":4967},{"class":736,"line":3927},[4968],{"type":48,"tag":734,"props":4969,"children":4970},{},[4971],{"type":54,"value":4972},"                        return rmsnorm(x, mod.weight, eps=epsilon)\n",{"type":48,"tag":734,"props":4974,"children":4975},{"class":736,"line":3935},[4976],{"type":48,"tag":734,"props":4977,"children":4978},{},[4979],{"type":54,"value":4980},"                    return forward\n",{"type":48,"tag":734,"props":4982,"children":4983},{"class":736,"line":3944},[4984],{"type":48,"tag":734,"props":4985,"children":4986},{},[4987],{"type":54,"value":4988},"                module.forward = make_forward(module, eps)\n",{"type":48,"tag":734,"props":4990,"children":4991},{"class":736,"line":3953},[4992],{"type":48,"tag":734,"props":4993,"children":4994},{},[4995],{"type":54,"value":4996},"            else:\n",{"type":48,"tag":734,"props":4998,"children":4999},{"class":736,"line":3962},[5000],{"type":48,"tag":734,"props":5001,"children":5002},{},[5003],{"type":54,"value":5004},"                def make_forward(epsilon):\n",{"type":48,"tag":734,"props":5006,"children":5007},{"class":736,"line":3971},[5008],{"type":48,"tag":734,"props":5009,"children":5010},{},[5011],{"type":54,"value":4964},{"type":48,"tag":734,"props":5013,"children":5014},{"class":736,"line":3980},[5015],{"type":48,"tag":734,"props":5016,"children":5017},{},[5018],{"type":54,"value":5019},"                        w = torch.ones(x.shape[-1], device=x.device, dtype=x.dtype)\n",{"type":48,"tag":734,"props":5021,"children":5022},{"class":736,"line":3988},[5023],{"type":48,"tag":734,"props":5024,"children":5025},{},[5026],{"type":54,"value":5027},"                        return rmsnorm(x, w, eps=epsilon)\n",{"type":48,"tag":734,"props":5029,"children":5030},{"class":736,"line":3997},[5031],{"type":48,"tag":734,"props":5032,"children":5033},{},[5034],{"type":54,"value":4980},{"type":48,"tag":734,"props":5036,"children":5037},{"class":736,"line":4006},[5038],{"type":48,"tag":734,"props":5039,"children":5040},{},[5041],{"type":54,"value":5042},"                module.forward = make_forward(eps)\n",{"type":48,"tag":734,"props":5044,"children":5045},{"class":736,"line":4015},[5046],{"type":48,"tag":734,"props":5047,"children":5048},{"emptyLinePlaceholder":41},[5049],{"type":54,"value":761},{"type":48,"tag":734,"props":5051,"children":5052},{"class":736,"line":4024},[5053],{"type":48,"tag":734,"props":5054,"children":5055},{},[5056],{"type":54,"value":5057},"# Usage\n",{"type":48,"tag":734,"props":5059,"children":5060},{"class":736,"line":4033},[5061],{"type":48,"tag":734,"props":5062,"children":5063},{},[5064],{"type":54,"value":5065},"pipe = LTXPipeline.from_pretrained(\"Lightricks\u002FLTX-Video\", torch_dtype=torch.bfloat16)\n",{"type":48,"tag":734,"props":5067,"children":5069},{"class":736,"line":5068},27,[5070],{"type":48,"tag":734,"props":5071,"children":5072},{},[5073],{"type":54,"value":4835},{"type":48,"tag":734,"props":5075,"children":5077},{"class":736,"line":5076},28,[5078],{"type":48,"tag":734,"props":5079,"children":5080},{},[5081],{"type":54,"value":5082},"patch_rmsnorm_modules(pipe.transformer)\n",{"type":48,"tag":734,"props":5084,"children":5086},{"class":736,"line":5085},29,[5087],{"type":48,"tag":734,"props":5088,"children":5089},{},[5090],{"type":54,"value":5091},"pipe.enable_model_cpu_offload()\n",{"type":48,"tag":78,"props":5093,"children":5095},{"id":5094},"kernel-specific-guidelines",[5096],{"type":54,"value":5097},"Kernel-Specific Guidelines",{"type":48,"tag":123,"props":5099,"children":5101},{"id":5100},"rmsnorm",[5102],{"type":54,"value":5103},"RMSNorm",{"type":48,"tag":1625,"props":5105,"children":5106},{},[5107,5118,5123,5138,5169,5179],{"type":48,"tag":961,"props":5108,"children":5109},{},[5110,5112],{"type":54,"value":5111},"Input layout: ",{"type":48,"tag":171,"props":5113,"children":5115},{"className":5114},[],[5116],{"type":54,"value":5117},"[..., hidden_size]",{"type":48,"tag":961,"props":5119,"children":5120},{},[5121],{"type":54,"value":5122},"Epsilon default: 1e-6",{"type":48,"tag":961,"props":5124,"children":5125},{},[5126,5131,5133],{"type":48,"tag":63,"props":5127,"children":5128},{},[5129],{"type":54,"value":5130},"Weight may be None",{"type":54,"value":5132}," if ",{"type":48,"tag":171,"props":5134,"children":5136},{"className":5135},[],[5137],{"type":54,"value":4420},{"type":48,"tag":961,"props":5139,"children":5140},{},[5141,5146,5148,5153,5155,5160,5162,5167],{"type":48,"tag":63,"props":5142,"children":5143},{},[5144],{"type":54,"value":5145},"Vectorization:",{"type":54,"value":5147}," Use ",{"type":48,"tag":171,"props":5149,"children":5151},{"className":5150},[],[5152],{"type":54,"value":2948},{"type":54,"value":5154}," for BF16, ",{"type":48,"tag":171,"props":5156,"children":5158},{"className":5157},[],[5159],{"type":54,"value":3050},{"type":54,"value":5161}," for FP16, ",{"type":48,"tag":171,"props":5163,"children":5165},{"className":5164},[],[5166],{"type":54,"value":3104},{"type":54,"value":5168}," for FP32",{"type":48,"tag":961,"props":5170,"children":5171},{},[5172,5177],{"type":48,"tag":63,"props":5173,"children":5174},{},[5175],{"type":54,"value":5176},"Performance:",{"type":54,"value":5178}," 2.67x faster than PyTorch with vectorized implementation",{"type":48,"tag":961,"props":5180,"children":5181},{},[5182,5187],{"type":48,"tag":63,"props":5183,"children":5184},{},[5185],{"type":54,"value":5186},"Bandwidth:",{"type":54,"value":5188}," Achieves ~38% of H100's 3.35 TB\u002Fs theoretical bandwidth",{"type":48,"tag":123,"props":5190,"children":5192},{"id":5191},"rope",[5193],{"type":54,"value":5194},"RoPE",{"type":48,"tag":1625,"props":5196,"children":5197},{},[5198,5211,5224],{"type":48,"tag":961,"props":5199,"children":5200},{},[5201,5203,5209],{"type":54,"value":5202},"1D: ",{"type":48,"tag":171,"props":5204,"children":5206},{"className":5205},[],[5207],{"type":54,"value":5208},"[batch, seq, heads, head_dim]",{"type":54,"value":5210}," - for text",{"type":48,"tag":961,"props":5212,"children":5213},{},[5214,5216,5222],{"type":54,"value":5215},"3D: ",{"type":48,"tag":171,"props":5217,"children":5219},{"className":5218},[],[5220],{"type":54,"value":5221},"[batch, t*h*w, heads, head_dim]",{"type":54,"value":5223}," - for video",{"type":48,"tag":961,"props":5225,"children":5226},{},[5227,5229],{"type":54,"value":5228},"LTX-Video computes its own RoPE via ",{"type":48,"tag":171,"props":5230,"children":5232},{"className":5231},[],[5233],{"type":54,"value":5234},"LTXVideoRotaryPosEmbed",{"type":48,"tag":123,"props":5236,"children":5238},{"id":5237},"geglu-vs-gelu",[5239],{"type":54,"value":5240},"GEGLU vs GELU",{"type":48,"tag":1625,"props":5242,"children":5243},{},[5244,5268,5278],{"type":48,"tag":961,"props":5245,"children":5246},{},[5247,5252,5254,5260,5262],{"type":48,"tag":63,"props":5248,"children":5249},{},[5250],{"type":54,"value":5251},"GEGLU",{"type":54,"value":5253},": Input ",{"type":48,"tag":171,"props":5255,"children":5257},{"className":5256},[],[5258],{"type":54,"value":5259},"[batch, seq, 2*hidden]",{"type":54,"value":5261}," -> Output ",{"type":48,"tag":171,"props":5263,"children":5265},{"className":5264},[],[5266],{"type":54,"value":5267},"[batch, seq, hidden]",{"type":48,"tag":961,"props":5269,"children":5270},{},[5271,5276],{"type":48,"tag":63,"props":5272,"children":5273},{},[5274],{"type":54,"value":5275},"GELU",{"type":54,"value":5277},": Standard activation",{"type":48,"tag":961,"props":5279,"children":5280},{},[5281],{"type":48,"tag":63,"props":5282,"children":5283},{},[5284],{"type":54,"value":5285},"LTX-Video uses GELU, NOT GEGLU",{"type":48,"tag":123,"props":5287,"children":5289},{"id":5288},"adaln",[5290],{"type":54,"value":5291},"AdaLN",{"type":48,"tag":1625,"props":5293,"children":5294},{},[5295,5306],{"type":48,"tag":961,"props":5296,"children":5297},{},[5298,5300],{"type":54,"value":5299},"Formula: ",{"type":48,"tag":171,"props":5301,"children":5303},{"className":5302},[],[5304],{"type":54,"value":5305},"norm(x) * weight * (1 + scale) + shift",{"type":48,"tag":961,"props":5307,"children":5308},{},[5309],{"type":54,"value":5310},"Used in DiT blocks for conditioning",{"type":48,"tag":78,"props":5312,"children":5314},{"id":5313},"performance-profiling",[5315],{"type":54,"value":5316},"Performance Profiling",{"type":48,"tag":723,"props":5318,"children":5320},{"className":1119,"code":5319,"language":1121,"meta":728,"style":728},"# NVIDIA Nsight Systems\nnsys profile -o profile python your_script.py\n\n# NVIDIA Nsight Compute\nncu --set full -o metrics python your_script.py\n",[5321],{"type":48,"tag":171,"props":5322,"children":5323},{"__ignoreMap":728},[5324,5332,5364,5371,5379],{"type":48,"tag":734,"props":5325,"children":5326},{"class":736,"line":737},[5327],{"type":48,"tag":734,"props":5328,"children":5329},{"style":1131},[5330],{"type":54,"value":5331},"# NVIDIA Nsight Systems\n",{"type":48,"tag":734,"props":5333,"children":5334},{"class":736,"line":746},[5335,5340,5345,5350,5354,5359],{"type":48,"tag":734,"props":5336,"children":5337},{"style":1140},[5338],{"type":54,"value":5339},"nsys",{"type":48,"tag":734,"props":5341,"children":5342},{"style":1145},[5343],{"type":54,"value":5344}," profile",{"type":48,"tag":734,"props":5346,"children":5347},{"style":1145},[5348],{"type":54,"value":5349}," -o",{"type":48,"tag":734,"props":5351,"children":5352},{"style":1145},[5353],{"type":54,"value":5344},{"type":48,"tag":734,"props":5355,"children":5356},{"style":1145},[5357],{"type":54,"value":5358}," python",{"type":48,"tag":734,"props":5360,"children":5361},{"style":1145},[5362],{"type":54,"value":5363}," your_script.py\n",{"type":48,"tag":734,"props":5365,"children":5366},{"class":736,"line":755},[5367],{"type":48,"tag":734,"props":5368,"children":5369},{"emptyLinePlaceholder":41},[5370],{"type":54,"value":761},{"type":48,"tag":734,"props":5372,"children":5373},{"class":736,"line":764},[5374],{"type":48,"tag":734,"props":5375,"children":5376},{"style":1131},[5377],{"type":54,"value":5378},"# NVIDIA Nsight Compute\n",{"type":48,"tag":734,"props":5380,"children":5381},{"class":736,"line":823},[5382,5387,5392,5397,5401,5406,5410],{"type":48,"tag":734,"props":5383,"children":5384},{"style":1140},[5385],{"type":54,"value":5386},"ncu",{"type":48,"tag":734,"props":5388,"children":5389},{"style":1145},[5390],{"type":54,"value":5391}," --set",{"type":48,"tag":734,"props":5393,"children":5394},{"style":1145},[5395],{"type":54,"value":5396}," full",{"type":48,"tag":734,"props":5398,"children":5399},{"style":1145},[5400],{"type":54,"value":5349},{"type":48,"tag":734,"props":5402,"children":5403},{"style":1145},[5404],{"type":54,"value":5405}," metrics",{"type":48,"tag":734,"props":5407,"children":5408},{"style":1145},[5409],{"type":54,"value":5358},{"type":48,"tag":734,"props":5411,"children":5412},{"style":1145},[5413],{"type":54,"value":5363},{"type":48,"tag":78,"props":5415,"children":5417},{"id":5416},"common-issues",[5418],{"type":54,"value":5419},"Common Issues",{"type":48,"tag":2900,"props":5421,"children":5422},{},[5423],{"type":48,"tag":57,"props":5424,"children":5425},{},[5426],{"type":48,"tag":63,"props":5427,"children":5428},{},[5429,5430,5436],{"type":54,"value":4081},{"type":48,"tag":90,"props":5431,"children":5433},{"href":5432},"references\u002Ftroubleshooting.md",[5434],{"type":54,"value":5435},"troubleshooting.md",{"type":54,"value":5437}," for all common issues and solutions.",{"type":48,"tag":57,"props":5439,"children":5440},{},[5441],{"type":54,"value":5442},"Quick fixes:",{"type":48,"tag":1625,"props":5444,"children":5445},{},[5446,5456,5473,5483,5499],{"type":48,"tag":961,"props":5447,"children":5448},{},[5449,5454],{"type":48,"tag":63,"props":5450,"children":5451},{},[5452],{"type":54,"value":5453},"\"NoneType has no attribute contiguous\"",{"type":54,"value":5455},": RMSNorm weight is None, create ones",{"type":48,"tag":961,"props":5457,"children":5458},{},[5459,5464,5466,5472],{"type":48,"tag":63,"props":5460,"children":5461},{},[5462],{"type":54,"value":5463},"isinstance() not matching",{"type":54,"value":5465},": Use ",{"type":48,"tag":171,"props":5467,"children":5469},{"className":5468},[],[5470],{"type":54,"value":5471},"type(module).__name__",{"type":54,"value":256},{"type":48,"tag":961,"props":5474,"children":5475},{},[5476,5481],{"type":48,"tag":63,"props":5477,"children":5478},{},[5479],{"type":54,"value":5480},"GEGLU not called",{"type":54,"value":5482},": Model uses GELU, not GEGLU",{"type":48,"tag":961,"props":5484,"children":5485},{},[5486,5491,5493],{"type":48,"tag":63,"props":5487,"children":5488},{},[5489],{"type":54,"value":5490},"Patching doesn't persist",{"type":54,"value":5492},": Inject before ",{"type":48,"tag":171,"props":5494,"children":5496},{"className":5495},[],[5497],{"type":54,"value":5498},"enable_model_cpu_offload()",{"type":48,"tag":961,"props":5500,"children":5501},{},[5502,5507],{"type":48,"tag":63,"props":5503,"children":5504},{},[5505],{"type":54,"value":5506},"torch.compile fails with custom kernels",{"type":54,"value":5508},": See below",{"type":48,"tag":123,"props":5510,"children":5512},{"id":5511},"torchcompile-compatibility",[5513],{"type":54,"value":5514},"torch.compile Compatibility",{"type":48,"tag":57,"props":5516,"children":5517},{},[5518,5520,5525,5526,5531],{"type":54,"value":5519},"Custom CUDA kernels and ",{"type":48,"tag":171,"props":5521,"children":5523},{"className":5522},[],[5524],{"type":54,"value":1752},{"type":54,"value":700},{"type":48,"tag":63,"props":5527,"children":5528},{},[5529],{"type":54,"value":5530},"mutually exclusive",{"type":54,"value":5532}," unless you register the kernel as a PyTorch custom op.",{"type":48,"tag":57,"props":5534,"children":5535},{},[5536],{"type":48,"tag":63,"props":5537,"children":5538},{},[5539],{"type":54,"value":5540},"Error message:",{"type":48,"tag":723,"props":5542,"children":5545},{"className":5543,"code":5544,"language":54},[2601],"torch._dynamo.exc.Unsupported: Attempted to call function marked as skipped\n",[5546],{"type":48,"tag":171,"props":5547,"children":5548},{"__ignoreMap":728},[5549],{"type":54,"value":5544},{"type":48,"tag":57,"props":5551,"children":5552},{},[5553],{"type":48,"tag":63,"props":5554,"children":5555},{},[5556],{"type":54,"value":5557},"Workaround options:",{"type":48,"tag":957,"props":5559,"children":5560},{},[5561,5579,5590],{"type":48,"tag":961,"props":5562,"children":5563},{},[5564,5565,5570,5572,5577],{"type":54,"value":4426},{"type":48,"tag":171,"props":5566,"children":5568},{"className":5567},[],[5569],{"type":54,"value":1975},{"type":54,"value":5571}," without ",{"type":48,"tag":171,"props":5573,"children":5575},{"className":5574},[],[5576],{"type":54,"value":2019},{"type":54,"value":5578}," (6% speedup)",{"type":48,"tag":961,"props":5580,"children":5581},{},[5582,5583,5588],{"type":54,"value":4426},{"type":48,"tag":171,"props":5584,"children":5586},{"className":5585},[],[5587],{"type":54,"value":2019},{"type":54,"value":5589}," without custom kernels (34% speedup)",{"type":48,"tag":961,"props":5591,"children":5592},{},[5593],{"type":54,"value":5594},"Add a fake\u002Fmeta implementation for the C++-registered op (see below)",{"type":48,"tag":57,"props":5596,"children":5597},{},[5598,5603,5605,5610,5612,5617,5619,5624],{"type":48,"tag":63,"props":5599,"children":5600},{},[5601],{"type":54,"value":5602},"To make the op torch.compile-compatible:",{"type":54,"value":5604}," ops registered via ",{"type":48,"tag":171,"props":5606,"children":5608},{"className":5607},[],[5609],{"type":54,"value":222},{"type":54,"value":5611}," in C++ are already proper custom ops — do NOT re-wrap them with ",{"type":48,"tag":171,"props":5613,"children":5615},{"className":5614},[],[5616],{"type":54,"value":650},{"type":54,"value":5618}," in Python. Just register a fake (meta) implementation using the generated ",{"type":48,"tag":171,"props":5620,"children":5622},{"className":5621},[],[5623],{"type":54,"value":698},{"type":54,"value":5625}," helpers:",{"type":48,"tag":723,"props":5627,"children":5629},{"className":889,"code":5628,"language":23,"meta":728,"style":728},"import torch\nfrom ._ops import ops, add_op_namespace_prefix\n\n@torch.library.register_fake(add_op_namespace_prefix(\"rmsnorm_forward\"))\ndef _(out, input, weight, eps):\n    return None  # out-variant op: no shape changes\n",[5630],{"type":48,"tag":171,"props":5631,"children":5632},{"__ignoreMap":728},[5633,5640,5648,5655,5663,5671],{"type":48,"tag":734,"props":5634,"children":5635},{"class":736,"line":737},[5636],{"type":48,"tag":734,"props":5637,"children":5638},{},[5639],{"type":54,"value":902},{"type":48,"tag":734,"props":5641,"children":5642},{"class":736,"line":746},[5643],{"type":48,"tag":734,"props":5644,"children":5645},{},[5646],{"type":54,"value":5647},"from ._ops import ops, add_op_namespace_prefix\n",{"type":48,"tag":734,"props":5649,"children":5650},{"class":736,"line":755},[5651],{"type":48,"tag":734,"props":5652,"children":5653},{"emptyLinePlaceholder":41},[5654],{"type":54,"value":761},{"type":48,"tag":734,"props":5656,"children":5657},{"class":736,"line":764},[5658],{"type":48,"tag":734,"props":5659,"children":5660},{},[5661],{"type":54,"value":5662},"@torch.library.register_fake(add_op_namespace_prefix(\"rmsnorm_forward\"))\n",{"type":48,"tag":734,"props":5664,"children":5665},{"class":736,"line":823},[5666],{"type":48,"tag":734,"props":5667,"children":5668},{},[5669],{"type":54,"value":5670},"def _(out, input, weight, eps):\n",{"type":48,"tag":734,"props":5672,"children":5673},{"class":736,"line":832},[5674],{"type":48,"tag":734,"props":5675,"children":5676},{},[5677],{"type":54,"value":5678},"    return None  # out-variant op: no shape changes\n",{"type":48,"tag":78,"props":5680,"children":5682},{"id":5681},"see-also",[5683],{"type":54,"value":5684},"See Also",{"type":48,"tag":123,"props":5686,"children":5688},{"id":5687},"scripts",[5689],{"type":54,"value":5690},"Scripts",{"type":48,"tag":1625,"props":5692,"children":5693},{},[5694,5710,5721,5732],{"type":48,"tag":961,"props":5695,"children":5696},{},[5697,5703,5705],{"type":48,"tag":90,"props":5698,"children":5700},{"href":5699},"scripts\u002Fbenchmark_example.py",[5701],{"type":54,"value":5702},"benchmark_example.py",{"type":54,"value":5704}," - ",{"type":48,"tag":63,"props":5706,"children":5707},{},[5708],{"type":54,"value":5709},"Benchmarking script for comparing optimized vs baseline - START HERE",{"type":48,"tag":961,"props":5711,"children":5712},{},[5713,5719],{"type":48,"tag":90,"props":5714,"children":5716},{"href":5715},"scripts\u002Fltx_kernel_injection_example.py",[5717],{"type":54,"value":5718},"ltx_kernel_injection_example.py",{"type":54,"value":5720}," - Minimal diffusers integration (~150 lines)",{"type":48,"tag":961,"props":5722,"children":5723},{},[5724,5730],{"type":48,"tag":90,"props":5725,"children":5727},{"href":5726},"scripts\u002Ftransformers_injection_example.py",[5728],{"type":54,"value":5729},"transformers_injection_example.py",{"type":54,"value":5731}," - Minimal transformers\u002FLLM integration (~120 lines)",{"type":48,"tag":961,"props":5733,"children":5734},{},[5735,5741],{"type":48,"tag":90,"props":5736,"children":5738},{"href":5737},"scripts\u002Fhuggingface_kernels_example.py",[5739],{"type":54,"value":5740},"huggingface_kernels_example.py",{"type":54,"value":5742}," - HuggingFace Kernels Hub integration",{"type":48,"tag":123,"props":5744,"children":5746},{"id":5745},"integration-guides",[5747],{"type":54,"value":5748},"Integration Guides",{"type":48,"tag":1625,"props":5750,"children":5751},{},[5752,5765,5774],{"type":48,"tag":961,"props":5753,"children":5754},{},[5755,5759,5760],{"type":48,"tag":90,"props":5756,"children":5757},{"href":4084},[5758],{"type":54,"value":4087},{"type":54,"value":5704},{"type":48,"tag":63,"props":5761,"children":5762},{},[5763],{"type":54,"value":5764},"HuggingFace Kernels Hub (get_kernel) - load pre-compiled kernels",{"type":48,"tag":961,"props":5766,"children":5767},{},[5768,5772],{"type":48,"tag":90,"props":5769,"children":5770},{"href":4364},[5771],{"type":54,"value":4367},{"type":54,"value":5773}," - Complete diffusers pipeline integration",{"type":48,"tag":961,"props":5775,"children":5776},{},[5777,5781],{"type":48,"tag":90,"props":5778,"children":5779},{"href":4387},[5780],{"type":54,"value":4390},{"type":54,"value":5782}," - Complete transformers\u002FLLM integration",{"type":48,"tag":123,"props":5784,"children":5786},{"id":5785},"gpu-optimization-guides",[5787],{"type":54,"value":5788},"GPU Optimization Guides",{"type":48,"tag":1625,"props":5790,"children":5791},{},[5792,5801,5810],{"type":48,"tag":961,"props":5793,"children":5794},{},[5795,5799],{"type":48,"tag":90,"props":5796,"children":5797},{"href":1565},[5798],{"type":54,"value":1568},{"type":54,"value":5800}," - H100 (Hopper, sm_90) deep dive",{"type":48,"tag":961,"props":5802,"children":5803},{},[5804,5808],{"type":48,"tag":90,"props":5805,"children":5806},{"href":1587},[5807],{"type":54,"value":1590},{"type":54,"value":5809}," - A100 (Ampere, sm_80) deep dive",{"type":48,"tag":961,"props":5811,"children":5812},{},[5813,5817],{"type":48,"tag":90,"props":5814,"children":5815},{"href":1609},[5816],{"type":54,"value":1612},{"type":54,"value":5818}," - T4 (Turing, sm_75) deep dive",{"type":48,"tag":123,"props":5820,"children":5822},{"id":5821},"reference",[5823],{"type":54,"value":2242},{"type":48,"tag":1625,"props":5825,"children":5826},{},[5827,5836,5847],{"type":48,"tag":961,"props":5828,"children":5829},{},[5830,5834],{"type":48,"tag":90,"props":5831,"children":5832},{"href":5432},[5833],{"type":54,"value":5435},{"type":54,"value":5835}," - Common issues and solutions",{"type":48,"tag":961,"props":5837,"children":5838},{},[5839,5845],{"type":48,"tag":90,"props":5840,"children":5842},{"href":5841},"references\u002Fkernel-templates.md",[5843],{"type":54,"value":5844},"kernel-templates.md",{"type":54,"value":5846}," - Complete kernel templates",{"type":48,"tag":961,"props":5848,"children":5849},{},[5850,5856],{"type":48,"tag":90,"props":5851,"children":5853},{"href":5852},"..\u002F..\u002F..\u002Fexamples\u002Fkernels\u002Frelu\u002F",[5854],{"type":54,"value":5855},"examples\u002Fkernels\u002Frelu\u002F",{"type":54,"value":5857}," - Canonical working kernel example (bindings, layers, tests)",{"type":48,"tag":123,"props":5859,"children":5861},{"id":5860},"external-resources",[5862],{"type":54,"value":5863},"External Resources",{"type":48,"tag":1625,"props":5865,"children":5866},{},[5867,5877,5886],{"type":48,"tag":961,"props":5868,"children":5869},{},[5870],{"type":48,"tag":90,"props":5871,"children":5874},{"href":5872,"rel":5873},"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fkernels\u002Fen\u002Findex",[93],[5875],{"type":54,"value":5876},"HuggingFace Kernels Documentation",{"type":48,"tag":961,"props":5878,"children":5879},{},[5880],{"type":48,"tag":90,"props":5881,"children":5883},{"href":28,"rel":5882},[93],[5884],{"type":54,"value":5885},"HuggingFace Kernels GitHub",{"type":48,"tag":961,"props":5887,"children":5888},{},[5889],{"type":48,"tag":90,"props":5890,"children":5893},{"href":5891,"rel":5892},"https:\u002F\u002Fhuggingface.co\u002Fkernels-community",[93],[5894],{"type":54,"value":5895},"Community Kernels on Hub",{"type":48,"tag":5897,"props":5898,"children":5899},"style",{},[5900],{"type":54,"value":5901},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5903,"total":764},[5904,5919,5927,5943],{"slug":5905,"name":5905,"fn":5906,"description":5907,"org":5908,"tags":5909,"stars":27,"repoUrl":28,"updatedAt":5918},"cpu-kernels","optimize C++ CPU kernels for Hugging Face","Provides guidance for writing, optimizing, and benchmarking C++ CPU kernels with SIMD intrinsics (AVX2\u002FAVX512) for the Hugging Face kernels ecosystem. Includes a two-phase workflow: Phase 1 correctness (generic → AVX2) and Phase 2 performance exploration (AVX512 with branching trial loop), runtime CPU dispatch, OpenMP threading, and brgemm integration for GEMM-heavy kernels.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5910,5913,5916,5917],{"name":5911,"slug":5912,"type":15},"C#","c",{"name":5914,"slug":5915,"type":15},"Engineering","engineering",{"name":9,"slug":17,"type":15},{"name":13,"slug":14,"type":15},"2026-06-09T07:19:37.783356",{"slug":4,"name":4,"fn":5,"description":6,"org":5920,"tags":5921,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5922,5923,5924,5925,5926],{"name":25,"slug":26,"type":15},{"name":19,"slug":20,"type":15},{"name":9,"slug":17,"type":15},{"name":13,"slug":14,"type":15},{"name":22,"slug":23,"type":15},{"slug":5928,"name":5928,"fn":5929,"description":5930,"org":5931,"tags":5932,"stars":27,"repoUrl":28,"updatedAt":5942},"rocm-kernels","build optimized Triton kernels for AMD GPUs","Provides guidance for writing and benchmarking optimized Triton kernels for AMD GPUs (MI355X, R9700) on ROCm, targeting HuggingFace diffusers (LTX-Video, SD3, FLUX) and transformers. Core kernels: RMSNorm, RoPE 3D, GEGLU, AdaLN. Includes XCD swizzle, autotune, diffusers integration patterns, and LTX-Video pipeline injection.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5933,5934,5935,5936,5939],{"name":25,"slug":26,"type":15},{"name":19,"slug":20,"type":15},{"name":13,"slug":14,"type":15},{"name":5937,"slug":5938,"type":15},"ROCm","rocm",{"name":5940,"slug":5941,"type":15},"Triton","triton","2026-04-16T05:06:54.179351",{"slug":5944,"name":5944,"fn":5945,"description":5946,"org":5947,"tags":5948,"stars":27,"repoUrl":28,"updatedAt":5953},"xpu-kernels","optimize Triton kernels for Intel XPU","Provides guidance for writing, optimizing, and benchmarking Triton kernels for Intel XPU GPUs (Battlemage\u002FArc Pro B50) using the Xe-Forge optimization framework. Includes an LLM-driven trial-loop workflow (analyze, validate, benchmark, profile, finalize), XPU-specific patterns (tensor descriptors, GRF mode, tile swizzling), KernelBench fused kernels, and Flash Attention.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5949,5950,5951,5952],{"name":25,"slug":26,"type":15},{"name":19,"slug":20,"type":15},{"name":13,"slug":14,"type":15},{"name":5940,"slug":5941,"type":15},"2026-07-18T05:15:18.767865",{"items":5955,"total":6121},[5956,5974,5988,6005,6018,6029,6042,6057,6071,6081,6094,6106],{"slug":5957,"name":5957,"fn":5958,"description":5959,"org":5960,"tags":5961,"stars":5971,"repoUrl":5972,"updatedAt":5973},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5962,5963,5964,5967,5968],{"name":19,"slug":20,"type":15},{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},"LLM","llm",{"name":22,"slug":23,"type":15},{"name":5969,"slug":5970,"type":15},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":5975,"name":5975,"fn":5976,"description":5977,"org":5978,"tags":5979,"stars":5985,"repoUrl":5986,"updatedAt":5987},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5980,5981,5982,5983,5984],{"name":25,"slug":26,"type":15},{"name":19,"slug":20,"type":15},{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},{"name":22,"slug":23,"type":15},18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828",{"slug":5989,"name":5989,"fn":5990,"description":5991,"org":5992,"tags":5993,"stars":6002,"repoUrl":6003,"updatedAt":6004},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5994,5997,6000,6001],{"name":5995,"slug":5996,"type":15},"CLI","cli",{"name":5998,"slug":5999,"type":15},"Datasets","datasets",{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-04-06T18:25:34.020855",{"slug":6006,"name":6006,"fn":6007,"description":6008,"org":6009,"tags":6010,"stars":6002,"repoUrl":6003,"updatedAt":6017},"hf-cloud-aws-context-discovery","discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6011,6014,6015],{"name":6012,"slug":6013,"type":15},"AWS","aws",{"name":5995,"slug":5996,"type":15},{"name":2190,"slug":6016,"type":15},"configuration","2026-07-08T05:55:33.716099",{"slug":6019,"name":6019,"fn":6020,"description":6021,"org":6022,"tags":6023,"stars":6002,"repoUrl":6003,"updatedAt":6028},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6024,6025,6026,6027],{"name":25,"slug":26,"type":15},{"name":6012,"slug":6013,"type":15},{"name":5914,"slug":5915,"type":15},{"name":22,"slug":23,"type":15},"2026-07-08T05:55:32.505017",{"slug":6030,"name":6030,"fn":6031,"description":6032,"org":6033,"tags":6034,"stars":6002,"repoUrl":6003,"updatedAt":6041},"hf-cloud-sagemaker-deployment-planner","plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6035,6036,6037,6040],{"name":25,"slug":26,"type":15},{"name":6012,"slug":6013,"type":15},{"name":6038,"slug":6039,"type":15},"Deployment","deployment",{"name":9,"slug":17,"type":15},"2026-07-08T05:55:37.387689",{"slug":6043,"name":6043,"fn":6044,"description":6045,"org":6046,"tags":6047,"stars":6002,"repoUrl":6003,"updatedAt":6056},"hf-cloud-sagemaker-iam-preflight","configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6048,6049,6050,6053],{"name":25,"slug":26,"type":15},{"name":6012,"slug":6013,"type":15},{"name":6051,"slug":6052,"type":15},"Permissions","permissions",{"name":6054,"slug":6055,"type":15},"Security","security","2026-07-08T05:55:34.948771",{"slug":6058,"name":6058,"fn":6059,"description":6060,"org":6061,"tags":6062,"stars":6002,"repoUrl":6003,"updatedAt":6070},"hf-cloud-sagemaker-production-defaults","create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6063,6064,6065,6066,6067],{"name":25,"slug":26,"type":15},{"name":6012,"slug":6013,"type":15},{"name":6038,"slug":6039,"type":15},{"name":9,"slug":17,"type":15},{"name":6068,"slug":6069,"type":15},"Monitoring","monitoring","2026-07-08T05:55:38.664702",{"slug":6072,"name":6072,"fn":6073,"description":6074,"org":6075,"tags":6076,"stars":6002,"repoUrl":6003,"updatedAt":6080},"hf-cloud-serving-image-selection","select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6077,6078,6079],{"name":25,"slug":26,"type":15},{"name":6012,"slug":6013,"type":15},{"name":6038,"slug":6039,"type":15},"2026-07-08T05:55:36.173465",{"slug":6082,"name":6082,"fn":6083,"description":6084,"org":6085,"tags":6086,"stars":6002,"repoUrl":6003,"updatedAt":6093},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6087,6088,6089,6090],{"name":5998,"slug":5999,"type":15},{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},{"name":6091,"slug":6092,"type":15},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":6095,"name":6095,"fn":6096,"description":6097,"org":6098,"tags":6099,"stars":6002,"repoUrl":6003,"updatedAt":6105},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6100,6101,6102,6103,6104],{"name":25,"slug":26,"type":15},{"name":5995,"slug":5996,"type":15},{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},{"name":13,"slug":14,"type":15},"2026-06-13T07:23:57.101435",{"slug":6107,"name":6107,"fn":6108,"description":6109,"org":6110,"tags":6111,"stars":6002,"repoUrl":6003,"updatedAt":6120},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6112,6115,6116,6117],{"name":6113,"slug":6114,"type":15},"Analytics","analytics",{"name":9,"slug":17,"type":15},{"name":5965,"slug":5966,"type":15},{"name":6118,"slug":6119,"type":15},"Research","research","2026-04-24T05:09:45.870658",37]