[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-cpu-kernels":3,"mdc--gdhumq-key":35,"related-repo-huggingface-cpu-kernels":5145,"related-org-huggingface-cpu-kernels":5198},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":33,"mdContent":34},"cpu-kernels","optimize C++ CPU kernels for Hugging Face","Provides guidance for writing, optimizing, and benchmarking C++ CPU kernels with SIMD intrinsics (AVX2\u002FAVX512) for the Hugging Face kernels ecosystem. Includes a two-phase workflow: Phase 1 correctness (generic → AVX2) and Phase 2 performance exploration (AVX512 with branching trial loop), runtime CPU dispatch, OpenMP threading, and brgemm integration for GEMM-heavy kernels.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,16,18,21],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":17,"type":15},"hugging-face",{"name":19,"slug":20,"type":15},"C#","c",{"name":22,"slug":23,"type":15},"Engineering","engineering",712,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels","2026-06-09T07:19:37.783356",null,112,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":32},[],"Build compute kernels and load them from the Hub.","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002FHEAD\u002Fkernel-builder\u002Fskills\u002Fcpu-kernels","---\nname: cpu-kernels\ndescription: \"Provides guidance for writing, optimizing, and benchmarking C++ CPU kernels with SIMD intrinsics (AVX2\u002FAVX512) for the Hugging Face kernels ecosystem. Includes a two-phase workflow: Phase 1 correctness (generic → AVX2) and Phase 2 performance exploration (AVX512 with branching trial loop), runtime CPU dispatch, OpenMP threading, and brgemm integration for GEMM-heavy kernels.\"\ndisable-model-invocation: false\nuser-invocable: true\nallowed-tools: \"Read, Grep, Glob, Bash\"\nargument-hint: \"kernel type: rmsnorm, flash-attention, quantized-gemm, activation, reduction, optimize, benchmark\"\n---\n\n# CPU C++ Kernels for x86 Processors\n\nThis skill provides patterns and guidance for developing optimized C++ kernels targeting x86 CPUs (Intel Xeon and compatible processors) with AVX2 and AVX512 intrinsics. Kernels are compiled via `kernel-builder` and distributed through the Hugging Face kernels ecosystem.\n\n> **Who runs these commands?** *You*, the agent — not a human. This is an autonomous loop: you write\u002Fedit the C++ kernel, build it, then run the scripts below as tools (via Bash) to check correctness, benchmark, and profile. You read each result, record it with `trial_manager.py`, decide the next change from the Phase 2 decision tree, and repeat until you hit `early_stop_speedup` or run all `max_trials`.\n\n## Key Concepts (read before the Quick Start)\n\nThe commands use a few names that mean different things. They are **not** interchangeable:\n\n| Name (example) | What it is | Used by |\n|----------------|-----------|---------|\n| **`baseline.py`** | The **PyTorch reference implementation** you optimize against. It is the ground truth for correctness *and* the speed reference for speedup. **It must define `get_inputs()`** and **either** `get_reference_output()` **or** a `Model` class (plus optional `get_init_inputs()`). You write this file (or it is given) before starting. | every script |\n| **`my_rmsnorm`** | A **trial-tree label** — an arbitrary name you pick for this optimization task. `trial_manager.py` stores all attempts under `trials\u002Fmy_rmsnorm\u002F`. It is *only* a tracking ID. | `trial_manager.py` only |\n| **`my_kernel`** | The **installed Python package name** — the build artifact produced by `kernel-builder build` + `pip install`. This is the importable module that contains your compiled kernel. | `--kernel-package` |\n| **`my_kernel.rms_norm`** | An **`\u003Cpackage>.\u003Cfunction>` path** — the actual callable inside the installed package. Passed to `--op` to tell the benchmark\u002Fprofiler which function to run. | `--op` |\n\n> ⚠️ **`--op` means two different things depending on the script.** In `analyze_op.py`, `--op` is a plain **operation name** (e.g. `\"rms_norm\"`) used to look up compute\u002Fmemory characteristics. In `benchmark_cpu.py` and `cpu_profiler.py`, `--op` is a **`package.function` path** (e.g. `my_kernel.rms_norm`) used to import and call your kernel. Same flag, different meaning — read each command below carefully.\n\n## Quick Start\n\n### Write a New CPU Kernel\n\nThe example below optimizes an RMSNorm kernel. The trial label is `my_rmsnorm`, the built package is `my_kernel`, and its function is `my_kernel.rms_norm` — keep these consistent across all six steps.\n\n```bash\n# 1. Analyze the target op. Here --op is an OPERATION NAME (looked up in the\n#    knowledge base), not a package path.\npython scripts\u002Fanalyze_op.py --op \"rms_norm\" --shapes \"1024x4096,2048x8192\"\n\n# 2. Initialize trial tracking. Args: \u003Ctrial-label> \u003Cbaseline-file>.\n#    Creates trials\u002Fmy_rmsnorm\u002F and records baseline.py as the reference.\npython scripts\u002Ftrial_manager.py init my_rmsnorm baseline.py\n\n# 3. Build the kernel package (produces the installable 'my_kernel' wheel).\ncd \u002Fpath\u002Fto\u002Fmy-kernel && kernel-builder build --release && pip install dist\u002F*.whl --force-reinstall\n\n# 4. Benchmark correctness + performance. Here --op is a PACKAGE.FUNCTION path.\n#    Compares my_kernel.rms_norm against baseline.py (correctness + speedup).\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package my_kernel --op my_kernel.rms_norm\n\n# 5. Profile with perf stat (same package.function path as step 4).\npython scripts\u002Fcpu_profiler.py --kernel-package my_kernel --op my_kernel.rms_norm\n\n# 6. Finalize: promote the best trial in trials\u002Fmy_rmsnorm\u002F into output\u002F.\npython scripts\u002Ftrial_manager.py finalize my_rmsnorm output\u002F\n```\n\n## Supported Hardware\n\n| ISA | Extensions | Key Instructions | Typical CPUs |\n|-----|-----------|-----------------|-------------|\n| **AVX2** | FMA, F16C | `_mm256_fmadd_ps`, `_mm256_cvtph_ps` | Most x86 CPUs (2013+) |\n| **AVX512** | F, BF16, VL, DQ, BW, VBMI | `_mm512_dpbf16_ps`, `_mm512_permutexvar_epi16` | Intel Xeon |\n\n### GEMM Acceleration: brgemm\n\nFor kernels that involve matrix multiplication (quantized GEMM, Flash Attention, MoE), large-M cases use `at::native::cpublas::brgemm()` — a PyTorch wrapper around oneDNN brgemm, which internally dispatches to AMX tile instructions on Intel Xeon (4th Gen+). Small-M cases (M ≤ 4 for bf16) fall back to hand-written `tinygemm` using AVX512 `_mm512_dpbf16_ps`. See [brgemm_patterns.yaml](references\u002Fbrgemm_patterns.yaml) for details.\n\n> **Note**: brgemm is NOT used in element-wise kernels (RMSNorm, activations, reductions). Those use AVX512 intrinsics directly.\n\n## When This Skill Applies\n\nUse this skill when:\n- Writing C++ CPU kernels with SIMD intrinsics for the HF kernels ecosystem\n- Optimizing existing CPU kernels (e.g., adding AVX512 to a generic implementation)\n- Implementing quantized GEMM kernels (INT4, NF4, FP4, FP8, MXFP4)\n- Implementing Flash Attention or other attention kernels for CPU\n- Building kernels with `kernel-builder` that target `backend = \"cpu\"`\n\n## Two-Phase Optimization Workflow\n\nCPU kernel development has two distinct phases with different strategies.\n\n### Configuration — Read `config.yaml` first\n\nAt the start of every session, read `scripts\u002Fconfig.yaml`. It controls:\n- **`max_trials`** — hard cap on Phase 2 optimization trials\n- **`early_stop_speedup`** — speedup vs PyTorch baseline to trigger early stop (default: 3.0)\n- **`perf_stat_enabled`** — if `true`, use `perf stat` for profiling (default)\n- **`vtune_enabled`** — if `true`, use VTune for detailed microarchitecture analysis\n- **`build_command`** — command to build the kernel package\n\n### Rules — Never Violate\n\n1. **ONLY modify** C++ kernel files (`.cpp`, `.hpp`), `torch_binding.cpp`, and `build.toml`. Do NOT create benchmark or test scripts.\n2. **NEVER write custom timing code** — ONLY use `scripts\u002Fbenchmark_cpu.py`.\n3. If a tool fails, **STOP and report the error**. Do NOT work around it with custom scripts.\n4. Generated kernels must follow the **runtime dispatch pattern** with `cpu_features.hpp` — see `references\u002Fruntime_dispatch.yaml`.\n5. Every kernel should have a **generic ATen fallback** that works on any CPU. If a specific path cannot have a meaningful fallback, use `TORCH_CHECK(false, ...)` with a clear error message.\n6. Each SIMD tier (AVX2, AVX512) must be in a **separate translation unit** (`.cpp` file) with its own compiler flags in `build.toml`. Do NOT mix intrinsics from different ISA levels in the same file.\n7. All SIMD implementations must handle **edge cases** (hidden_size not divisible by vector width).\n8. AVX2 tier is **optional** — most CPU kernels go directly from generic fallback to AVX512. Only add AVX2 when it provides meaningful benefit for element-wise ops.\n9. You **MUST run all `max_trials` trials** in Phase 2. Do NOT stop early due to plateau — the only valid early stop is speedup > `early_stop_speedup`.\n\n### Mandatory Tools\n\n| Tool | Command | Purpose |\n|------|---------|---------|\n| **Analyze** | `python scripts\u002Fanalyze_op.py --op \u003Cop_name> --shapes \u003Cshapes>` | Analyze PyTorch op: compute\u002Fmemory characteristics, SIMD strategy recommendations |\n| **Validate** | `python scripts\u002Fvalidate_cpu_kernel.py \u003Ckernel_dir>` | Static checks: alignment, OpenMP usage, intrinsics correctness, build.toml validation |\n| **Build** | `kernel-builder build --release` | Compile C++ kernel via build.toml into a wheel |\n| **Benchmark** | `python scripts\u002Fbenchmark_cpu.py \u003Cbaseline_file> --kernel-package \u003Cpkg> --op \u003Cfunc>` | Correctness + performance via `torch.utils.benchmark` |\n| **Profile** | `python scripts\u002Fcpu_profiler.py --kernel-package \u003Cpkg> --op \u003Cfunc>` | `perf stat` hardware counters + optimization recommendations |\n| **Trial Manager** | `python scripts\u002Ftrial_manager.py \u003Ccommand> ...` | Trial tree management (init\u002Fsave\u002Fresult\u002Fstatus\u002Fbest\u002Ffinalize) |\n\n> **Benchmark discipline**: Pin to a single NUMA node — `numactl --cpunodebind=0 --membind=0 python scripts\u002Fbenchmark_cpu.py ...`. See [threading_patterns.yaml](references\u002Fthreading_patterns.yaml).\n\n\n### Phase 1: Correctness (Linear, No Branching)\n\nBuild the kernel tier by tier. Each tier must be correct before moving on.\n\n#### Tier 0: Generic Fallback\n- Implement using **PyTorch ATen ops only** (no intrinsics).\n- This serves as the portable baseline that runs on any CPU.\n- Must produce results matching the PyTorch reference within tolerance.\n- File: `\u003Ckernel>_cpu\u002F\u003Ckernel>_cpu.cpp`\n\n```bash\n# Validate + build\npython scripts\u002Fvalidate_cpu_kernel.py .\nkernel-builder build --release\npip install dist\u002F*.whl --force-reinstall\n\n# Benchmark (this also establishes the PyTorch baseline time)\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package my_kernel --op my_kernel.rms_norm\n```\n\n#### Tier 1: AVX2 (Optional)\n- Add AVX2 implementation using `_mm256_*` intrinsics.\n- Compile with `-mavx2 -mfma -mf16c -fopenmp`.\n- Must be correct; performance improvement is a bonus.\n- File: `\u003Ckernel>_cpu\u002F\u003Ckernel>_avx2.cpp`\n\n#### Tier 2: AVX512\n- Add AVX512 implementation using `_mm512_*` intrinsics.\n- Compile with `-mavx512f -mavx512bf16 -mavx512vl -mavx512dq -mavx512bw -mavx512vbmi -mfma -mf16c -fopenmp`.\n- For GEMM kernels using brgemm, additionally add `-mamx-tile -mamx-bf16 -mamx-int8`.\n- This is the **entry point to Phase 2** — performance optimization starts here.\n- File: `\u003Ckernel>_cpu\u002F\u003Ckernel>_avx512.cpp`\n\n### Phase 2: Performance Exploration (Branching, With Backtracking)\n\nOnce AVX512 is correct, optimize for peak performance. This phase uses the trial manager with branching.\n\n```bash\n# Initialize Phase 2 trials\npython scripts\u002Ftrial_manager.py init \u003Ckernel_name> baseline.py\n\n# For each trial:\n# 1. Modify kernel code (AVX512 intrinsics, or brgemm for GEMM kernels)\n# 2. Build\nkernel-builder build --release && pip install dist\u002F*.whl --force-reinstall\n# 3. Benchmark\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n# 4. Save trial\npython scripts\u002Ftrial_manager.py save \u003Ckernel_name> \u003Cdir> --parent \u003Cparent_id> --strategy \"description\"\n# 5. Record result\npython scripts\u002Ftrial_manager.py result \u003Ckernel_name> \u003Ctrial_id> --correctness pass --speedup \u003Cfloat> --baseline_us \u003Cfloat> --kernel_us \u003Cfloat>\n# 6. Profile (after t1, or when plateaued)\npython scripts\u002Fcpu_profiler.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n```\n\n#### Phase 2 Decision Tree\n\n| Condition | Action |\n|-----------|--------|\n| **Speedup > `early_stop_speedup`** | Stop — excellent result (the only valid early stop) |\n| **Speedup improved** | Continue on this branch, try next optimization |\n| **Speedup regressed** | Branch back to best trial, try different strategy |\n| **Correctness failed** | Fix on same branch (usually alignment or SIMD boundary bug) |\n| **After t1 (if `perf_stat_enabled`)** | Run `cpu_profiler.py` — mandatory first profile |\n| **IPC \u003C 1.0** | Memory bound → add prefetch, change cache blocking |\n| **L1 miss rate high** | Tile too large for L1 → reduce tile size |\n| **L3 miss rate high** | Working set too large → add cache blocking |\n| **Plateau after 2+ trials** | Do NOT keep tuning the same knobs. Change the **approach**: switch algorithm path (tinygemm ↔ brgemm), change the fusion\u002Fblocking\u002Fdata-layout strategy, or reconsider the dispatch heuristic. A different structure beats endless parameter sweeps. |\n| **Max trials reached** | Stop — must run all `max_trials` from `config.yaml` |\n\n#### Optimization Search Space (Phase 2)\n\n> These tables are a **starting menu of values seen in existing kernels, not an exhaustive recipe**. Use them to seed trials, but when a branch plateaus, prefer a structurally different idea (algorithm, fusion, memory strategy) over sweeping these knobs further. See the try-harder tree in [optimization_levels.yaml](references\u002Foptimization_levels.yaml).\n\n**GEMM kernels (quantized GEMM, Flash Attention, MoE):**\n\n| Dimension | Actual Values in Existing Kernels | Notes |\n|-----------|----------------------------------|-------|\n| BLOCK_M (tinygemm path) | 4 | Small M, fused dequant+GEMM |\n| BLOCK_M (brgemm path) | 32 (= 2×TILE_M) | Must be multiple of TILE_M=16 |\n| BLOCK_N (tinygemm) | 32, 64 | Determines register tile COLS = BLOCK_N\u002F16 |\n| BLOCK_N (brgemm) | 32 (= 2×TILE_N) | Must be multiple of TILE_N=16 |\n| BLOCK_K | 128 (= 4×TILE_K) | K-dimension blocking |\n| BLOCK_M\u002FN (flash-attn2) | 256 \u002F 768 | Much larger — attention-specific |\n| K-loop unroll | 4 (`#pragma GCC unroll 4`) | All GEMM kernels use 4 |\n| Prefetch distance | 0 (disabled), 64 elements ahead | L1 prefetch via `_MM_HINT_T0` |\n| Algorithm path | `use_brgemm` threshold (e.g. M > 4) | Switch from tinygemm to brgemm |\n| brgemm dequant policy | `use_brgemm_dequant_out` (e.g. M > 100)| True=pre-dequant all B upfront; False=dequant per K-block |\n| L2 cache budget | 1 MB (50% of 2 MB L2) | Controls N-blocking in `loop_2d` |\n| Thread decomposition | 2D factorization: nth_m × nth_n | Based on M\u002FN aspect ratio |\n\n**Element-wise kernels (RMSNorm, activations):**\n\n| Dimension | Actual Values | Notes |\n|-----------|---------------|-------|\n| Vectorization width | 16 (fp32), 32 (bf16) | Per-type VEC_ELEM_NUM |\n| Prefetch hint | `_MM_HINT_T1` (L2) | Different from GEMM (L1) |\n| OpenMP grain size | 1024 | `at::parallel_for` grain |\n| Threading | `#pragma omp parallel for` over rows | Simple 1D parallelism |\n\n### Phase 2 Finalization\n\n```bash\npython scripts\u002Ftrial_manager.py finalize \u003Ckernel_name> output\u002F\n# Re-run benchmark without cached baseline for final accurate comparison\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n```\n\n## Reference Docs — Read During Phase 1\n\n| Doc | Contents |\n|-----|----------|\n| `references\u002Fruntime_dispatch.yaml` | cpu_features.hpp pattern, dispatch tiers |\n| `references\u002Fbuild_system.yaml` | build.toml multi-target CPU compilation |\n| `references\u002Fimplementation_reference.md` | C++ kernel templates, Unroll\\\u003CN\\>, tinygemm, torch_binding.cpp |\n| `references\u002Fcorrectness.yaml` | Critical constraints: alignment, FTZ\u002FDAZ, denormals |\n\n## Reference Docs — Read During Phase 2\n\n| Doc | Contents |\n|-----|----------|\n| `references\u002Fsimd_optimization_patterns.yaml` | AVX2\u002FAVX512 vector abstractions and patterns |\n| `references\u002Fquantized_gemm_patterns.yaml` | LUT + tinygemm + Unroll template for 4-bit GEMM |\n| `references\u002Fbrgemm_patterns.yaml` | brgemm API usage, VNNI packing, tinygemm vs brgemm selection (GEMM kernels only) |\n| `references\u002Fmemory_patterns.yaml` | Prefetch, alignment, cache blocking |\n| `references\u002Fthreading_patterns.yaml` | OpenMP parallel patterns |\n| `references\u002Fdtype_optimizations.yaml` | bf16\u002Ffp8\u002Fint8 handling and conversion on CPU |\n| `references\u002Foptimization_levels.yaml` | Progressive L1→L5 optimization checklist + try-harder tree |\n| `references\u002Foptimization_strategies.md` | Strategy reference, decision tree, checklist |\n| `references\u002Fworkflow_details.md` | Detailed trial loop workflow |\n| `references\u002Fhuggingface-kernels-integration.md` | Hub integration for CPU kernels |\n\n## Core CPU Kernel Patterns\n\n### Runtime Dispatch (Required for All Kernels)\n\nEvery CPU kernel has its own `cpu_features.hpp` (in its own namespace) and dispatches at runtime. Most kernels dispatch as AVX512 → fallback (no AVX2 tier):\n\n```cpp\n\u002F\u002F my_kernel_cpu\u002Fcpu_features.hpp — each kernel has its OWN copy\nnamespace my_kernel_cpu {\nclass CPUFeatures {\npublic:\n    static bool hasAVX512BF16() { \u002F* CPUID + XCR0 checks *\u002F }\n    static bool hasAVX2() { \u002F* CPUID check *\u002F }\n    \u002F\u002F GEMM kernels also check: static bool hasAMX() { ... }\n};\n}\n\n\u002F\u002F my_kernel_cpu\u002Fmy_kernel_cpu.cpp — dispatcher\n#include \"cpu_features.hpp\"\n#include \"my_kernel_avx512.hpp\"\n\nvoid my_kernel(torch::Tensor& out, const torch::Tensor& input, ...) {\n    if (CPUFeatures::hasAVX512BF16()) {\n        avx512::my_kernel_impl(out, input, ...);\n    } else {\n        \u002F\u002F ATen fallback — inline or in a separate _fallback.cpp\n        out = torch::some_aten_op(input, ...);\n    }\n}\n```\n\n> **Note**: Only rmsnorm has a three-tier dispatch (AVX512 → AVX2 → ATen). GEMM kernels skip AVX2. Flash-attn2 additionally requires AMX via `hasAllRequiredFeatures()`.\n\n> Full pattern: [runtime_dispatch.yaml](references\u002Fruntime_dispatch.yaml)\n\n### build.toml Multi-Target Compilation\n\nEach SIMD tier is a separate `[kernel.*]` section with its own compiler flags. The `include` directive is required for header resolution:\n\n```toml\n[kernel.my_kernel_cpu]\nbackend = \"cpu\"\ndepends = [\"torch\"]\ninclude = [\"my_kernel_cpu\"]\nsrc = [\n    \"my_kernel_cpu\u002Fmy_kernel_cpu.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_cpu_torch.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_cpu.hpp\",\n    \"my_kernel_cpu\u002Fcpu_features.hpp\",\n]\n\n[kernel.my_kernel_cpu_avx512]\nbackend = \"cpu\"\n# Note: For GEMM kernels (e.g., flash-attn2, megablocks), you must also include \"-mamx-tile\", \"-mamx-bf16\", \"-mamx-int8\"\ncxx-flags = [\"-mavx512f\", \"-mavx512bf16\", \"-mavx512vl\", \"-mavx512dq\", \"-mavx512bw\", \"-mavx512vbmi\", \"-mfma\", \"-mf16c\", \"-fopenmp\"]\ndepends = [\"torch\"]\ninclude = [\"my_kernel_cpu\"]\nsrc = [\n    \"my_kernel_cpu\u002Fmy_kernel_avx512.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_avx512.hpp\",\n]\n```\n\n> **Note**: Every section needs `include = [\"\u003Ckernel_dir>\"]` for header resolution. The `_torch.cpp` file bridges Python-facing declarations to the C++ dispatcher. AVX2 section is optional (only rmsnorm has one).\n\n> Full pattern: [build_system.yaml](references\u002Fbuild_system.yaml)\n\n### torch_binding.cpp Registration\n\nAll kernels use `registration.h` macros for op registration:\n\n```cpp\n#include \"registration.h\"\n\n\u002F\u002F Forward declarations\n#if defined(CPU_KERNEL)\ntorch::Tensor my_kernel_cpu_forward(torch::Tensor input, torch::Tensor weight, float eps);\n#endif\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n    ops.def(\"forward(Tensor input, Tensor weight, float eps) -> Tensor\");\n    ops.impl(\"forward\", torch::kCPU, &my_kernel_cpu_forward);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n```\n\n> **Note**: `registration.h` is provided by kernel-builder. Multi-device kernels (rmsnorm, megablocks) use `#if defined(CPU_KERNEL)` \u002F `#elif defined(CUDA_KERNEL)` guards.\n\n### Vector Type Abstractions (AVX512)\n\nWrap raw intrinsics in typed vector classes for readability:\n\n```cpp\n\u002F\u002F cpu_types_avx512.hpp\nstruct FP32Vec16 {\n    __m512 reg;\n    FP32Vec16(float v) : reg(_mm512_set1_ps(v)) {}\n    FP32Vec16(__m512 r) : reg(r) {}\n    FP32Vec16 operator*(const FP32Vec16& other) const {\n        return FP32Vec16(_mm512_mul_ps(reg, other.reg));\n    }\n    float reduce_sum() const { return _mm512_reduce_add_ps(reg); }\n};\n```\n\n> Full pattern: [simd_optimization_patterns.yaml](references\u002Fsimd_optimization_patterns.yaml)\n\n### Quantized GEMM Template (INT4\u002FNF4\u002FFP4)\n\nAll 4-bit quantized GEMM kernels share the same skeleton — only the LUT and zero-point handling differ:\n\n```\nnibble split → zero subtract → LUT lookup → _mm512_dpbf16_ps accumulate → scale fmadd (per group) → bf16 output\n```\n\nThe parameterized components:\n- **LUT**: GPTQ (linear INT4), BnB (NF4\u002FFP4), MegaBlocks (FP8\u002FMXFP4)\n- **Zero-point**: per-group (GPTQ), none\u002Fencoded in LUT (BnB), per-block (FP8)\n- **Algorithm**: tinygemm (small M, fused) vs brgemm (large M, unpack+BLAS)\n- **Weight conversion**: The C++ kernel expects a specific block-interleaved format, NOT raw checkpoint format. Each framework converts in its own repo:\n  - **GPTQ**: `transform_cpu()` unpacks int32→uint8, reorders by g_idx, transposes to [N,K]; then `convert_weight_packed_zp()` repacks to [N,K\u002F2] block-interleaved (BLOCK_N=32). Zeros unpacked to [groups,N] uint8. Scales to bf16. Done at first forward in GPTQModel repo.\n  - **BnB**: `_convert_weight_packed_for_cpu()` unpacks uint8 nibbles→[N,K], repacks to [N,K\u002F2] block-interleaved (same algo as GPTQ). Denests nested absmax. Transposes scales to [K\u002Fblocksize,N] bf16. Done at first forward in bitsandbytes repo.\n  - **Megablocks MoE**: `ops.convert_weight_packed()` does transpose+VNNI pack. `ops.convert_scale_packed()` reorders scales. Cached via `packed_weight=True`.\n- **VNNI Conversion (K\u002FV Activations)**:\n  - **Flash Attention**: `pack_vnni()` per tile per forward (K\u002FV change every call, so caching is not possible).\n- **Element-wise (RMSNorm)**: No conversion needed.\n\n> Full pattern: [quantized_gemm_patterns.yaml](references\u002Fquantized_gemm_patterns.yaml), weight conversion: [brgemm_patterns.yaml](references\u002Fbrgemm_patterns.yaml)\n\n## Critical CPU Constraints\n\n- **Always use unaligned loads**: All existing kernels use `_mm512_loadu_*` exclusively. Never use `_mm512_load_*`.\n- **Edge cases**: When `hidden_size % VEC_ELEM_NUM != 0`, handle the tail with scalar or masked SIMD ops.\n- **FTZ\u002FDAZ**: Flush-to-zero and denormals-as-zero may be set by PyTorch. Do NOT assume IEEE 754 denormal behavior.\n- **OpenMP overhead**: For small tensors, use `adjust_num_threads(m)` to reduce thread count. GEMM kernels use `parallel_2d` for 2D thread decomposition.\n- **bf16 precision**: `_mm512_dpbf16_ps` accumulates in fp32 but inputs are bf16 — precision loss is expected. Use atol=1e-2 for correctness checks.\n- **Data alignment**: Use `alignas(64)` for stack-allocated tile buffers to optimize cache-line access.\n\n> Full constraint list: [correctness.yaml](references\u002Fcorrectness.yaml)\n\n## Common Issues\n\n| Issue | Symptom | Fix |\n|-------|---------|-----|\n| **Unaligned access** | SEGFAULT or wrong results | Use `_mm512_loadu_*` instead of `_mm512_load_*` |\n| **Missing tail handling** | Wrong results for non-aligned sizes | Add scalar loop for remainder elements |\n| **OpenMP on small tensor** | Slower than baseline | Add `if (num_tokens > threshold)` guard |\n| **Wrong compiler flags** | Intrinsics not recognized | Check build.toml `cxx-flags` matches code |\n| **Silent scalar `at::vec`** | Kernel ~2x slow, no error; `objdump` shows 0 `%zmm` \u002F `nm` shows `expf@GLIBC` | Define `CPU_CAPABILITY_AVX512` for TUs using `at::vec::Vectorized` (see build_system.yaml) |\n| **CPUID detection wrong** | Crashes on older CPU | Verify `cpu_features.hpp` checks OS support (XCR0) |\n\n## Project Structure\n\n```\ncpu-kernels\u002F\n├── SKILL.md                                    # This file (skill definition + workflow)\n├── manifest.txt                                # Files included in this skill\n│\n├── scripts\u002F                                    # Standalone CLI tools\n│   ├── analyze_op.py                           # PyTorch op → compute\u002Fmemory analysis\n│   ├── validate_cpu_kernel.py                  # Static checks on C++ kernel code\n│   ├── benchmark_cpu.py                        # Correctness + performance via torch.utils.benchmark\n│   ├── cpu_profiler.py                         # perf stat hardware counters + recommendations\n│   ├── trial_manager.py                        # Tree-structured trial management\n│   ├── config.yaml                             # Session config (max_trials, profiler, build)\n│   └── config.py                               # Shared configuration loader\n│\n└── references\u002F                                 # Knowledge base\n    ├── correctness.yaml                        # Critical constraints for CPU kernels\n    ├── runtime_dispatch.yaml                   # cpu_features.hpp + dispatch pattern\n    ├── build_system.yaml                       # build.toml multi-target CPU compilation\n    ├── simd_optimization_patterns.yaml         # AVX2\u002FAVX512 vector abstractions and patterns\n    ├── quantized_gemm_patterns.yaml            # LUT + tinygemm\u002Fbrgemm template\n    ├── brgemm_patterns.yaml                     # brgemm API, VNNI packing, tinygemm fallback (GEMM kernels only)\n    ├── memory_patterns.yaml                    # Prefetch, alignment, cache blocking\n    ├── threading_patterns.yaml                 # OpenMP parallel patterns\n    ├── dtype_optimizations.yaml                # bf16\u002Ffp8\u002Fint8 handling on CPU\n    ├── optimization_levels.yaml                # Progressive L1→L5 optimization checklist\n    ├── implementation_reference.md             # C++ kernel templates and examples\n    ├── optimization_strategies.md              # Strategy reference + decision tree\n    ├── workflow_details.md                     # Detailed workflow reference\n    └── huggingface-kernels-integration.md      # HF kernels ecosystem integration guide\n```\n\n## See Also\n\n### Tools\n- [analyze_op.py](scripts\u002Fanalyze_op.py) — Analyze PyTorch op characteristics\n- [validate_cpu_kernel.py](scripts\u002Fvalidate_cpu_kernel.py) — Static kernel validation\n- [benchmark_cpu.py](scripts\u002Fbenchmark_cpu.py) — Correctness + performance measurement\n- [cpu_profiler.py](scripts\u002Fcpu_profiler.py) — perf stat hardware counters\n- [trial_manager.py](scripts\u002Ftrial_manager.py) — Trial tree management\n\n### CPU Optimization References\n- [correctness.yaml](references\u002Fcorrectness.yaml) — Critical constraints\n- [simd_optimization_patterns.yaml](references\u002Fsimd_optimization_patterns.yaml) — SIMD patterns\n- [quantized_gemm_patterns.yaml](references\u002Fquantized_gemm_patterns.yaml) — Quantized GEMM template\n- [optimization_levels.yaml](references\u002Foptimization_levels.yaml) — Progressive optimization\n- [implementation_reference.md](references\u002Fimplementation_reference.md) — Code templates\n\n### External Resources\n- [Hugging Face Kernels](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels) — Kernel hub and builder CLI\n- [Intel Intrinsics Guide](https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fdocs\u002Fintrinsics-guide\u002F)\n- [kernel-builder Documentation](https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002Fmain\u002Fdocs)\n- [xpu-kernels skill](..\u002Fxpu-kernels\u002FSKILL.md) — the Intel XPU Triton skill this workflow was adapted from\n- [Xe-Forge](https:\u002F\u002Fgithub.com\u002FIntelLabs\u002FXe-Forge) — the LLM-driven optimization framework the skill methodology originates from\n\n## Acknowledgments\n\nThe methodology of this skill — the YAML knowledge base, the benchmark\u002Fvalidation harnesses, and the branching trial-manager optimization loop — was adapted from the [xpu-kernels skill](..\u002Fxpu-kernels\u002FSKILL.md) built by a group of Intel AI researchers, the IntelLabs team behind [Xe-Forge](https:\u002F\u002Fgithub.com\u002FIntelLabs\u002FXe-Forge), where the methodology originates. Thanks to the original authors for a solid foundation to build on.\n",{"data":36,"body":41},{"name":4,"description":6,"disable-model-invocation":37,"user-invocable":38,"allowed-tools":39,"argument-hint":40},false,true,"Read, Grep, Glob, Bash","kernel type: rmsnorm, flash-attention, quantized-gemm, activation, reduction, optimize, benchmark",{"type":42,"children":43},"root",[44,53,68,115,122,134,414,503,509,516,542,924,930,1036,1042,1079,1092,1098,1103,1146,1152,1157,1171,1184,1277,1283,1479,1485,1673,1700,1706,1711,1718,1754,1876,1882,1925,1931,1991,1997,2002,2487,2493,2713,2719,2741,2749,3023,3031,3146,3152,3257,3263,3352,3358,3545,3551,3557,3569,3753,3772,3785,3791,3812,3983,4011,4023,4029,4042,4149,4184,4190,4195,4280,4292,4298,4303,4313,4318,4518,4536,4542,4658,4671,4677,4916,4922,4931,4937,4943,4996,5002,5051,5057,5115,5121,5139],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"cpu-c-kernels-for-x86-processors",[50],{"type":51,"value":52},"text","CPU C++ Kernels for x86 Processors",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57,59,66],{"type":51,"value":58},"This skill provides patterns and guidance for developing optimized C++ kernels targeting x86 CPUs (Intel Xeon and compatible processors) with AVX2 and AVX512 intrinsics. Kernels are compiled via ",{"type":45,"tag":60,"props":61,"children":63},"code",{"className":62},[],[64],{"type":51,"value":65},"kernel-builder",{"type":51,"value":67}," and distributed through the Hugging Face kernels ecosystem.",{"type":45,"tag":69,"props":70,"children":71},"blockquote",{},[72],{"type":45,"tag":54,"props":73,"children":74},{},[75,81,83,89,91,97,99,105,107,113],{"type":45,"tag":76,"props":77,"children":78},"strong",{},[79],{"type":51,"value":80},"Who runs these commands?",{"type":51,"value":82}," ",{"type":45,"tag":84,"props":85,"children":86},"em",{},[87],{"type":51,"value":88},"You",{"type":51,"value":90},", the agent — not a human. This is an autonomous loop: you write\u002Fedit the C++ kernel, build it, then run the scripts below as tools (via Bash) to check correctness, benchmark, and profile. You read each result, record it with ",{"type":45,"tag":60,"props":92,"children":94},{"className":93},[],[95],{"type":51,"value":96},"trial_manager.py",{"type":51,"value":98},", decide the next change from the Phase 2 decision tree, and repeat until you hit ",{"type":45,"tag":60,"props":100,"children":102},{"className":101},[],[103],{"type":51,"value":104},"early_stop_speedup",{"type":51,"value":106}," or run all ",{"type":45,"tag":60,"props":108,"children":110},{"className":109},[],[111],{"type":51,"value":112},"max_trials",{"type":51,"value":114},".",{"type":45,"tag":116,"props":117,"children":119},"h2",{"id":118},"key-concepts-read-before-the-quick-start",[120],{"type":51,"value":121},"Key Concepts (read before the Quick Start)",{"type":45,"tag":54,"props":123,"children":124},{},[125,127,132],{"type":51,"value":126},"The commands use a few names that mean different things. They are ",{"type":45,"tag":76,"props":128,"children":129},{},[130],{"type":51,"value":131},"not",{"type":51,"value":133}," interchangeable:",{"type":45,"tag":135,"props":136,"children":137},"table",{},[138,162],{"type":45,"tag":139,"props":140,"children":141},"thead",{},[142],{"type":45,"tag":143,"props":144,"children":145},"tr",{},[146,152,157],{"type":45,"tag":147,"props":148,"children":149},"th",{},[150],{"type":51,"value":151},"Name (example)",{"type":45,"tag":147,"props":153,"children":154},{},[155],{"type":51,"value":156},"What it is",{"type":45,"tag":147,"props":158,"children":159},{},[160],{"type":51,"value":161},"Used by",{"type":45,"tag":163,"props":164,"children":165},"tbody",{},[166,255,314,365],{"type":45,"tag":143,"props":167,"children":168},{},[169,182,250],{"type":45,"tag":170,"props":171,"children":172},"td",{},[173],{"type":45,"tag":76,"props":174,"children":175},{},[176],{"type":45,"tag":60,"props":177,"children":179},{"className":178},[],[180],{"type":51,"value":181},"baseline.py",{"type":45,"tag":170,"props":183,"children":184},{},[185,187,192,194,199,201,212,214,219,220,226,227,232,234,240,242,248],{"type":51,"value":186},"The ",{"type":45,"tag":76,"props":188,"children":189},{},[190],{"type":51,"value":191},"PyTorch reference implementation",{"type":51,"value":193}," you optimize against. It is the ground truth for correctness ",{"type":45,"tag":84,"props":195,"children":196},{},[197],{"type":51,"value":198},"and",{"type":51,"value":200}," the speed reference for speedup. ",{"type":45,"tag":76,"props":202,"children":203},{},[204,206],{"type":51,"value":205},"It must define ",{"type":45,"tag":60,"props":207,"children":209},{"className":208},[],[210],{"type":51,"value":211},"get_inputs()",{"type":51,"value":213}," and ",{"type":45,"tag":76,"props":215,"children":216},{},[217],{"type":51,"value":218},"either",{"type":51,"value":82},{"type":45,"tag":60,"props":221,"children":223},{"className":222},[],[224],{"type":51,"value":225},"get_reference_output()",{"type":51,"value":82},{"type":45,"tag":76,"props":228,"children":229},{},[230],{"type":51,"value":231},"or",{"type":51,"value":233}," a ",{"type":45,"tag":60,"props":235,"children":237},{"className":236},[],[238],{"type":51,"value":239},"Model",{"type":51,"value":241}," class (plus optional ",{"type":45,"tag":60,"props":243,"children":245},{"className":244},[],[246],{"type":51,"value":247},"get_init_inputs()",{"type":51,"value":249},"). You write this file (or it is given) before starting.",{"type":45,"tag":170,"props":251,"children":252},{},[253],{"type":51,"value":254},"every script",{"type":45,"tag":143,"props":256,"children":257},{},[258,270,304],{"type":45,"tag":170,"props":259,"children":260},{},[261],{"type":45,"tag":76,"props":262,"children":263},{},[264],{"type":45,"tag":60,"props":265,"children":267},{"className":266},[],[268],{"type":51,"value":269},"my_rmsnorm",{"type":45,"tag":170,"props":271,"children":272},{},[273,275,280,282,287,289,295,297,302],{"type":51,"value":274},"A ",{"type":45,"tag":76,"props":276,"children":277},{},[278],{"type":51,"value":279},"trial-tree label",{"type":51,"value":281}," — an arbitrary name you pick for this optimization task. ",{"type":45,"tag":60,"props":283,"children":285},{"className":284},[],[286],{"type":51,"value":96},{"type":51,"value":288}," stores all attempts under ",{"type":45,"tag":60,"props":290,"children":292},{"className":291},[],[293],{"type":51,"value":294},"trials\u002Fmy_rmsnorm\u002F",{"type":51,"value":296},". It is ",{"type":45,"tag":84,"props":298,"children":299},{},[300],{"type":51,"value":301},"only",{"type":51,"value":303}," a tracking ID.",{"type":45,"tag":170,"props":305,"children":306},{},[307,312],{"type":45,"tag":60,"props":308,"children":310},{"className":309},[],[311],{"type":51,"value":96},{"type":51,"value":313}," only",{"type":45,"tag":143,"props":315,"children":316},{},[317,329,356],{"type":45,"tag":170,"props":318,"children":319},{},[320],{"type":45,"tag":76,"props":321,"children":322},{},[323],{"type":45,"tag":60,"props":324,"children":326},{"className":325},[],[327],{"type":51,"value":328},"my_kernel",{"type":45,"tag":170,"props":330,"children":331},{},[332,333,338,340,346,348,354],{"type":51,"value":186},{"type":45,"tag":76,"props":334,"children":335},{},[336],{"type":51,"value":337},"installed Python package name",{"type":51,"value":339}," — the build artifact produced by ",{"type":45,"tag":60,"props":341,"children":343},{"className":342},[],[344],{"type":51,"value":345},"kernel-builder build",{"type":51,"value":347}," + ",{"type":45,"tag":60,"props":349,"children":351},{"className":350},[],[352],{"type":51,"value":353},"pip install",{"type":51,"value":355},". This is the importable module that contains your compiled kernel.",{"type":45,"tag":170,"props":357,"children":358},{},[359],{"type":45,"tag":60,"props":360,"children":362},{"className":361},[],[363],{"type":51,"value":364},"--kernel-package",{"type":45,"tag":143,"props":366,"children":367},{},[368,380,406],{"type":45,"tag":170,"props":369,"children":370},{},[371],{"type":45,"tag":76,"props":372,"children":373},{},[374],{"type":45,"tag":60,"props":375,"children":377},{"className":376},[],[378],{"type":51,"value":379},"my_kernel.rms_norm",{"type":45,"tag":170,"props":381,"children":382},{},[383,385,396,398,404],{"type":51,"value":384},"An ",{"type":45,"tag":76,"props":386,"children":387},{},[388,394],{"type":45,"tag":60,"props":389,"children":391},{"className":390},[],[392],{"type":51,"value":393},"\u003Cpackage>.\u003Cfunction>",{"type":51,"value":395}," path",{"type":51,"value":397}," — the actual callable inside the installed package. Passed to ",{"type":45,"tag":60,"props":399,"children":401},{"className":400},[],[402],{"type":51,"value":403},"--op",{"type":51,"value":405}," to tell the benchmark\u002Fprofiler which function to run.",{"type":45,"tag":170,"props":407,"children":408},{},[409],{"type":45,"tag":60,"props":410,"children":412},{"className":411},[],[413],{"type":51,"value":403},{"type":45,"tag":69,"props":415,"children":416},{},[417],{"type":45,"tag":54,"props":418,"children":419},{},[420,422,432,434,440,442,447,449,454,456,462,464,470,471,477,478,483,485,495,496,501],{"type":51,"value":421},"⚠️ ",{"type":45,"tag":76,"props":423,"children":424},{},[425,430],{"type":45,"tag":60,"props":426,"children":428},{"className":427},[],[429],{"type":51,"value":403},{"type":51,"value":431}," means two different things depending on the script.",{"type":51,"value":433}," In ",{"type":45,"tag":60,"props":435,"children":437},{"className":436},[],[438],{"type":51,"value":439},"analyze_op.py",{"type":51,"value":441},", ",{"type":45,"tag":60,"props":443,"children":445},{"className":444},[],[446],{"type":51,"value":403},{"type":51,"value":448}," is a plain ",{"type":45,"tag":76,"props":450,"children":451},{},[452],{"type":51,"value":453},"operation name",{"type":51,"value":455}," (e.g. ",{"type":45,"tag":60,"props":457,"children":459},{"className":458},[],[460],{"type":51,"value":461},"\"rms_norm\"",{"type":51,"value":463},") used to look up compute\u002Fmemory characteristics. In ",{"type":45,"tag":60,"props":465,"children":467},{"className":466},[],[468],{"type":51,"value":469},"benchmark_cpu.py",{"type":51,"value":213},{"type":45,"tag":60,"props":472,"children":474},{"className":473},[],[475],{"type":51,"value":476},"cpu_profiler.py",{"type":51,"value":441},{"type":45,"tag":60,"props":479,"children":481},{"className":480},[],[482],{"type":51,"value":403},{"type":51,"value":484}," is a ",{"type":45,"tag":76,"props":486,"children":487},{},[488,494],{"type":45,"tag":60,"props":489,"children":491},{"className":490},[],[492],{"type":51,"value":493},"package.function",{"type":51,"value":395},{"type":51,"value":455},{"type":45,"tag":60,"props":497,"children":499},{"className":498},[],[500],{"type":51,"value":379},{"type":51,"value":502},") used to import and call your kernel. Same flag, different meaning — read each command below carefully.",{"type":45,"tag":116,"props":504,"children":506},{"id":505},"quick-start",[507],{"type":51,"value":508},"Quick Start",{"type":45,"tag":510,"props":511,"children":513},"h3",{"id":512},"write-a-new-cpu-kernel",[514],{"type":51,"value":515},"Write a New CPU Kernel",{"type":45,"tag":54,"props":517,"children":518},{},[519,521,526,528,533,535,540],{"type":51,"value":520},"The example below optimizes an RMSNorm kernel. The trial label is ",{"type":45,"tag":60,"props":522,"children":524},{"className":523},[],[525],{"type":51,"value":269},{"type":51,"value":527},", the built package is ",{"type":45,"tag":60,"props":529,"children":531},{"className":530},[],[532],{"type":51,"value":328},{"type":51,"value":534},", and its function is ",{"type":45,"tag":60,"props":536,"children":538},{"className":537},[],[539],{"type":51,"value":379},{"type":51,"value":541}," — keep these consistent across all six steps.",{"type":45,"tag":543,"props":544,"children":549},"pre",{"className":545,"code":546,"language":547,"meta":548,"style":548},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# 1. Analyze the target op. Here --op is an OPERATION NAME (looked up in the\n#    knowledge base), not a package path.\npython scripts\u002Fanalyze_op.py --op \"rms_norm\" --shapes \"1024x4096,2048x8192\"\n\n# 2. Initialize trial tracking. Args: \u003Ctrial-label> \u003Cbaseline-file>.\n#    Creates trials\u002Fmy_rmsnorm\u002F and records baseline.py as the reference.\npython scripts\u002Ftrial_manager.py init my_rmsnorm baseline.py\n\n# 3. Build the kernel package (produces the installable 'my_kernel' wheel).\ncd \u002Fpath\u002Fto\u002Fmy-kernel && kernel-builder build --release && pip install dist\u002F*.whl --force-reinstall\n\n# 4. Benchmark correctness + performance. Here --op is a PACKAGE.FUNCTION path.\n#    Compares my_kernel.rms_norm against baseline.py (correctness + speedup).\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package my_kernel --op my_kernel.rms_norm\n\n# 5. Profile with perf stat (same package.function path as step 4).\npython scripts\u002Fcpu_profiler.py --kernel-package my_kernel --op my_kernel.rms_norm\n\n# 6. Finalize: promote the best trial in trials\u002Fmy_rmsnorm\u002F into output\u002F.\npython scripts\u002Ftrial_manager.py finalize my_rmsnorm output\u002F\n","bash","",[550],{"type":45,"tag":60,"props":551,"children":552},{"__ignoreMap":548},[553,565,574,630,639,648,657,685,693,702,772,780,789,798,835,843,852,881,889,898],{"type":45,"tag":554,"props":555,"children":558},"span",{"class":556,"line":557},"line",1,[559],{"type":45,"tag":554,"props":560,"children":562},{"style":561},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[563],{"type":51,"value":564},"# 1. Analyze the target op. Here --op is an OPERATION NAME (looked up in the\n",{"type":45,"tag":554,"props":566,"children":568},{"class":556,"line":567},2,[569],{"type":45,"tag":554,"props":570,"children":571},{"style":561},[572],{"type":51,"value":573},"#    knowledge base), not a package path.\n",{"type":45,"tag":554,"props":575,"children":577},{"class":556,"line":576},3,[578,584,590,595,601,606,611,616,620,625],{"type":45,"tag":554,"props":579,"children":581},{"style":580},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[582],{"type":51,"value":583},"python",{"type":45,"tag":554,"props":585,"children":587},{"style":586},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[588],{"type":51,"value":589}," scripts\u002Fanalyze_op.py",{"type":45,"tag":554,"props":591,"children":592},{"style":586},[593],{"type":51,"value":594}," --op",{"type":45,"tag":554,"props":596,"children":598},{"style":597},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[599],{"type":51,"value":600}," \"",{"type":45,"tag":554,"props":602,"children":603},{"style":586},[604],{"type":51,"value":605},"rms_norm",{"type":45,"tag":554,"props":607,"children":608},{"style":597},[609],{"type":51,"value":610},"\"",{"type":45,"tag":554,"props":612,"children":613},{"style":586},[614],{"type":51,"value":615}," --shapes",{"type":45,"tag":554,"props":617,"children":618},{"style":597},[619],{"type":51,"value":600},{"type":45,"tag":554,"props":621,"children":622},{"style":586},[623],{"type":51,"value":624},"1024x4096,2048x8192",{"type":45,"tag":554,"props":626,"children":627},{"style":597},[628],{"type":51,"value":629},"\"\n",{"type":45,"tag":554,"props":631,"children":633},{"class":556,"line":632},4,[634],{"type":45,"tag":554,"props":635,"children":636},{"emptyLinePlaceholder":38},[637],{"type":51,"value":638},"\n",{"type":45,"tag":554,"props":640,"children":642},{"class":556,"line":641},5,[643],{"type":45,"tag":554,"props":644,"children":645},{"style":561},[646],{"type":51,"value":647},"# 2. Initialize trial tracking. Args: \u003Ctrial-label> \u003Cbaseline-file>.\n",{"type":45,"tag":554,"props":649,"children":651},{"class":556,"line":650},6,[652],{"type":45,"tag":554,"props":653,"children":654},{"style":561},[655],{"type":51,"value":656},"#    Creates trials\u002Fmy_rmsnorm\u002F and records baseline.py as the reference.\n",{"type":45,"tag":554,"props":658,"children":660},{"class":556,"line":659},7,[661,665,670,675,680],{"type":45,"tag":554,"props":662,"children":663},{"style":580},[664],{"type":51,"value":583},{"type":45,"tag":554,"props":666,"children":667},{"style":586},[668],{"type":51,"value":669}," scripts\u002Ftrial_manager.py",{"type":45,"tag":554,"props":671,"children":672},{"style":586},[673],{"type":51,"value":674}," init",{"type":45,"tag":554,"props":676,"children":677},{"style":586},[678],{"type":51,"value":679}," my_rmsnorm",{"type":45,"tag":554,"props":681,"children":682},{"style":586},[683],{"type":51,"value":684}," baseline.py\n",{"type":45,"tag":554,"props":686,"children":688},{"class":556,"line":687},8,[689],{"type":45,"tag":554,"props":690,"children":691},{"emptyLinePlaceholder":38},[692],{"type":51,"value":638},{"type":45,"tag":554,"props":694,"children":696},{"class":556,"line":695},9,[697],{"type":45,"tag":554,"props":698,"children":699},{"style":561},[700],{"type":51,"value":701},"# 3. Build the kernel package (produces the installable 'my_kernel' wheel).\n",{"type":45,"tag":554,"props":703,"children":705},{"class":556,"line":704},10,[706,712,717,722,727,732,737,741,746,751,756,762,767],{"type":45,"tag":554,"props":707,"children":709},{"style":708},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[710],{"type":51,"value":711},"cd",{"type":45,"tag":554,"props":713,"children":714},{"style":586},[715],{"type":51,"value":716}," \u002Fpath\u002Fto\u002Fmy-kernel",{"type":45,"tag":554,"props":718,"children":719},{"style":597},[720],{"type":51,"value":721}," &&",{"type":45,"tag":554,"props":723,"children":724},{"style":580},[725],{"type":51,"value":726}," kernel-builder",{"type":45,"tag":554,"props":728,"children":729},{"style":586},[730],{"type":51,"value":731}," build",{"type":45,"tag":554,"props":733,"children":734},{"style":586},[735],{"type":51,"value":736}," --release",{"type":45,"tag":554,"props":738,"children":739},{"style":597},[740],{"type":51,"value":721},{"type":45,"tag":554,"props":742,"children":743},{"style":580},[744],{"type":51,"value":745}," pip",{"type":45,"tag":554,"props":747,"children":748},{"style":586},[749],{"type":51,"value":750}," install",{"type":45,"tag":554,"props":752,"children":753},{"style":586},[754],{"type":51,"value":755}," dist\u002F",{"type":45,"tag":554,"props":757,"children":759},{"style":758},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[760],{"type":51,"value":761},"*",{"type":45,"tag":554,"props":763,"children":764},{"style":586},[765],{"type":51,"value":766},".whl",{"type":45,"tag":554,"props":768,"children":769},{"style":586},[770],{"type":51,"value":771}," --force-reinstall\n",{"type":45,"tag":554,"props":773,"children":775},{"class":556,"line":774},11,[776],{"type":45,"tag":554,"props":777,"children":778},{"emptyLinePlaceholder":38},[779],{"type":51,"value":638},{"type":45,"tag":554,"props":781,"children":783},{"class":556,"line":782},12,[784],{"type":45,"tag":554,"props":785,"children":786},{"style":561},[787],{"type":51,"value":788},"# 4. Benchmark correctness + performance. Here --op is a PACKAGE.FUNCTION path.\n",{"type":45,"tag":554,"props":790,"children":792},{"class":556,"line":791},13,[793],{"type":45,"tag":554,"props":794,"children":795},{"style":561},[796],{"type":51,"value":797},"#    Compares my_kernel.rms_norm against baseline.py (correctness + speedup).\n",{"type":45,"tag":554,"props":799,"children":801},{"class":556,"line":800},14,[802,806,811,816,821,826,830],{"type":45,"tag":554,"props":803,"children":804},{"style":580},[805],{"type":51,"value":583},{"type":45,"tag":554,"props":807,"children":808},{"style":586},[809],{"type":51,"value":810}," scripts\u002Fbenchmark_cpu.py",{"type":45,"tag":554,"props":812,"children":813},{"style":586},[814],{"type":51,"value":815}," baseline.py",{"type":45,"tag":554,"props":817,"children":818},{"style":586},[819],{"type":51,"value":820}," --kernel-package",{"type":45,"tag":554,"props":822,"children":823},{"style":586},[824],{"type":51,"value":825}," my_kernel",{"type":45,"tag":554,"props":827,"children":828},{"style":586},[829],{"type":51,"value":594},{"type":45,"tag":554,"props":831,"children":832},{"style":586},[833],{"type":51,"value":834}," my_kernel.rms_norm\n",{"type":45,"tag":554,"props":836,"children":838},{"class":556,"line":837},15,[839],{"type":45,"tag":554,"props":840,"children":841},{"emptyLinePlaceholder":38},[842],{"type":51,"value":638},{"type":45,"tag":554,"props":844,"children":846},{"class":556,"line":845},16,[847],{"type":45,"tag":554,"props":848,"children":849},{"style":561},[850],{"type":51,"value":851},"# 5. Profile with perf stat (same package.function path as step 4).\n",{"type":45,"tag":554,"props":853,"children":855},{"class":556,"line":854},17,[856,860,865,869,873,877],{"type":45,"tag":554,"props":857,"children":858},{"style":580},[859],{"type":51,"value":583},{"type":45,"tag":554,"props":861,"children":862},{"style":586},[863],{"type":51,"value":864}," scripts\u002Fcpu_profiler.py",{"type":45,"tag":554,"props":866,"children":867},{"style":586},[868],{"type":51,"value":820},{"type":45,"tag":554,"props":870,"children":871},{"style":586},[872],{"type":51,"value":825},{"type":45,"tag":554,"props":874,"children":875},{"style":586},[876],{"type":51,"value":594},{"type":45,"tag":554,"props":878,"children":879},{"style":586},[880],{"type":51,"value":834},{"type":45,"tag":554,"props":882,"children":884},{"class":556,"line":883},18,[885],{"type":45,"tag":554,"props":886,"children":887},{"emptyLinePlaceholder":38},[888],{"type":51,"value":638},{"type":45,"tag":554,"props":890,"children":892},{"class":556,"line":891},19,[893],{"type":45,"tag":554,"props":894,"children":895},{"style":561},[896],{"type":51,"value":897},"# 6. Finalize: promote the best trial in trials\u002Fmy_rmsnorm\u002F into output\u002F.\n",{"type":45,"tag":554,"props":899,"children":901},{"class":556,"line":900},20,[902,906,910,915,919],{"type":45,"tag":554,"props":903,"children":904},{"style":580},[905],{"type":51,"value":583},{"type":45,"tag":554,"props":907,"children":908},{"style":586},[909],{"type":51,"value":669},{"type":45,"tag":554,"props":911,"children":912},{"style":586},[913],{"type":51,"value":914}," finalize",{"type":45,"tag":554,"props":916,"children":917},{"style":586},[918],{"type":51,"value":679},{"type":45,"tag":554,"props":920,"children":921},{"style":586},[922],{"type":51,"value":923}," output\u002F\n",{"type":45,"tag":116,"props":925,"children":927},{"id":926},"supported-hardware",[928],{"type":51,"value":929},"Supported Hardware",{"type":45,"tag":135,"props":931,"children":932},{},[933,959],{"type":45,"tag":139,"props":934,"children":935},{},[936],{"type":45,"tag":143,"props":937,"children":938},{},[939,944,949,954],{"type":45,"tag":147,"props":940,"children":941},{},[942],{"type":51,"value":943},"ISA",{"type":45,"tag":147,"props":945,"children":946},{},[947],{"type":51,"value":948},"Extensions",{"type":45,"tag":147,"props":950,"children":951},{},[952],{"type":51,"value":953},"Key Instructions",{"type":45,"tag":147,"props":955,"children":956},{},[957],{"type":51,"value":958},"Typical CPUs",{"type":45,"tag":163,"props":960,"children":961},{},[962,999],{"type":45,"tag":143,"props":963,"children":964},{},[965,973,978,994],{"type":45,"tag":170,"props":966,"children":967},{},[968],{"type":45,"tag":76,"props":969,"children":970},{},[971],{"type":51,"value":972},"AVX2",{"type":45,"tag":170,"props":974,"children":975},{},[976],{"type":51,"value":977},"FMA, F16C",{"type":45,"tag":170,"props":979,"children":980},{},[981,987,988],{"type":45,"tag":60,"props":982,"children":984},{"className":983},[],[985],{"type":51,"value":986},"_mm256_fmadd_ps",{"type":51,"value":441},{"type":45,"tag":60,"props":989,"children":991},{"className":990},[],[992],{"type":51,"value":993},"_mm256_cvtph_ps",{"type":45,"tag":170,"props":995,"children":996},{},[997],{"type":51,"value":998},"Most x86 CPUs (2013+)",{"type":45,"tag":143,"props":1000,"children":1001},{},[1002,1010,1015,1031],{"type":45,"tag":170,"props":1003,"children":1004},{},[1005],{"type":45,"tag":76,"props":1006,"children":1007},{},[1008],{"type":51,"value":1009},"AVX512",{"type":45,"tag":170,"props":1011,"children":1012},{},[1013],{"type":51,"value":1014},"F, BF16, VL, DQ, BW, VBMI",{"type":45,"tag":170,"props":1016,"children":1017},{},[1018,1024,1025],{"type":45,"tag":60,"props":1019,"children":1021},{"className":1020},[],[1022],{"type":51,"value":1023},"_mm512_dpbf16_ps",{"type":51,"value":441},{"type":45,"tag":60,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":51,"value":1030},"_mm512_permutexvar_epi16",{"type":45,"tag":170,"props":1032,"children":1033},{},[1034],{"type":51,"value":1035},"Intel Xeon",{"type":45,"tag":510,"props":1037,"children":1039},{"id":1038},"gemm-acceleration-brgemm",[1040],{"type":51,"value":1041},"GEMM Acceleration: brgemm",{"type":45,"tag":54,"props":1043,"children":1044},{},[1045,1047,1053,1055,1061,1063,1068,1070,1077],{"type":51,"value":1046},"For kernels that involve matrix multiplication (quantized GEMM, Flash Attention, MoE), large-M cases use ",{"type":45,"tag":60,"props":1048,"children":1050},{"className":1049},[],[1051],{"type":51,"value":1052},"at::native::cpublas::brgemm()",{"type":51,"value":1054}," — a PyTorch wrapper around oneDNN brgemm, which internally dispatches to AMX tile instructions on Intel Xeon (4th Gen+). Small-M cases (M ≤ 4 for bf16) fall back to hand-written ",{"type":45,"tag":60,"props":1056,"children":1058},{"className":1057},[],[1059],{"type":51,"value":1060},"tinygemm",{"type":51,"value":1062}," using AVX512 ",{"type":45,"tag":60,"props":1064,"children":1066},{"className":1065},[],[1067],{"type":51,"value":1023},{"type":51,"value":1069},". See ",{"type":45,"tag":1071,"props":1072,"children":1074},"a",{"href":1073},"references\u002Fbrgemm_patterns.yaml",[1075],{"type":51,"value":1076},"brgemm_patterns.yaml",{"type":51,"value":1078}," for details.",{"type":45,"tag":69,"props":1080,"children":1081},{},[1082],{"type":45,"tag":54,"props":1083,"children":1084},{},[1085,1090],{"type":45,"tag":76,"props":1086,"children":1087},{},[1088],{"type":51,"value":1089},"Note",{"type":51,"value":1091},": brgemm is NOT used in element-wise kernels (RMSNorm, activations, reductions). Those use AVX512 intrinsics directly.",{"type":45,"tag":116,"props":1093,"children":1095},{"id":1094},"when-this-skill-applies",[1096],{"type":51,"value":1097},"When This Skill Applies",{"type":45,"tag":54,"props":1099,"children":1100},{},[1101],{"type":51,"value":1102},"Use this skill when:",{"type":45,"tag":1104,"props":1105,"children":1106},"ul",{},[1107,1113,1118,1123,1128],{"type":45,"tag":1108,"props":1109,"children":1110},"li",{},[1111],{"type":51,"value":1112},"Writing C++ CPU kernels with SIMD intrinsics for the HF kernels ecosystem",{"type":45,"tag":1108,"props":1114,"children":1115},{},[1116],{"type":51,"value":1117},"Optimizing existing CPU kernels (e.g., adding AVX512 to a generic implementation)",{"type":45,"tag":1108,"props":1119,"children":1120},{},[1121],{"type":51,"value":1122},"Implementing quantized GEMM kernels (INT4, NF4, FP4, FP8, MXFP4)",{"type":45,"tag":1108,"props":1124,"children":1125},{},[1126],{"type":51,"value":1127},"Implementing Flash Attention or other attention kernels for CPU",{"type":45,"tag":1108,"props":1129,"children":1130},{},[1131,1133,1138,1140],{"type":51,"value":1132},"Building kernels with ",{"type":45,"tag":60,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":51,"value":65},{"type":51,"value":1139}," that target ",{"type":45,"tag":60,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":51,"value":1145},"backend = \"cpu\"",{"type":45,"tag":116,"props":1147,"children":1149},{"id":1148},"two-phase-optimization-workflow",[1150],{"type":51,"value":1151},"Two-Phase Optimization Workflow",{"type":45,"tag":54,"props":1153,"children":1154},{},[1155],{"type":51,"value":1156},"CPU kernel development has two distinct phases with different strategies.",{"type":45,"tag":510,"props":1158,"children":1160},{"id":1159},"configuration-read-configyaml-first",[1161,1163,1169],{"type":51,"value":1162},"Configuration — Read ",{"type":45,"tag":60,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":51,"value":1168},"config.yaml",{"type":51,"value":1170}," first",{"type":45,"tag":54,"props":1172,"children":1173},{},[1174,1176,1182],{"type":51,"value":1175},"At the start of every session, read ",{"type":45,"tag":60,"props":1177,"children":1179},{"className":1178},[],[1180],{"type":51,"value":1181},"scripts\u002Fconfig.yaml",{"type":51,"value":1183},". It controls:",{"type":45,"tag":1104,"props":1185,"children":1186},{},[1187,1200,1213,1243,1263],{"type":45,"tag":1108,"props":1188,"children":1189},{},[1190,1198],{"type":45,"tag":76,"props":1191,"children":1192},{},[1193],{"type":45,"tag":60,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":51,"value":112},{"type":51,"value":1199}," — hard cap on Phase 2 optimization trials",{"type":45,"tag":1108,"props":1201,"children":1202},{},[1203,1211],{"type":45,"tag":76,"props":1204,"children":1205},{},[1206],{"type":45,"tag":60,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":51,"value":104},{"type":51,"value":1212}," — speedup vs PyTorch baseline to trigger early stop (default: 3.0)",{"type":45,"tag":1108,"props":1214,"children":1215},{},[1216,1225,1227,1233,1235,1241],{"type":45,"tag":76,"props":1217,"children":1218},{},[1219],{"type":45,"tag":60,"props":1220,"children":1222},{"className":1221},[],[1223],{"type":51,"value":1224},"perf_stat_enabled",{"type":51,"value":1226}," — if ",{"type":45,"tag":60,"props":1228,"children":1230},{"className":1229},[],[1231],{"type":51,"value":1232},"true",{"type":51,"value":1234},", use ",{"type":45,"tag":60,"props":1236,"children":1238},{"className":1237},[],[1239],{"type":51,"value":1240},"perf stat",{"type":51,"value":1242}," for profiling (default)",{"type":45,"tag":1108,"props":1244,"children":1245},{},[1246,1255,1256,1261],{"type":45,"tag":76,"props":1247,"children":1248},{},[1249],{"type":45,"tag":60,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":51,"value":1254},"vtune_enabled",{"type":51,"value":1226},{"type":45,"tag":60,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":51,"value":1232},{"type":51,"value":1262},", use VTune for detailed microarchitecture analysis",{"type":45,"tag":1108,"props":1264,"children":1265},{},[1266,1275],{"type":45,"tag":76,"props":1267,"children":1268},{},[1269],{"type":45,"tag":60,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":51,"value":1274},"build_command",{"type":51,"value":1276}," — command to build the kernel package",{"type":45,"tag":510,"props":1278,"children":1280},{"id":1279},"rules-never-violate",[1281],{"type":51,"value":1282},"Rules — Never Violate",{"type":45,"tag":1284,"props":1285,"children":1286},"ol",{},[1287,1328,1345,1357,1384,1404,1430,1442,1454],{"type":45,"tag":1108,"props":1288,"children":1289},{},[1290,1295,1297,1303,1304,1310,1312,1318,1320,1326],{"type":45,"tag":76,"props":1291,"children":1292},{},[1293],{"type":51,"value":1294},"ONLY modify",{"type":51,"value":1296}," C++ kernel files (",{"type":45,"tag":60,"props":1298,"children":1300},{"className":1299},[],[1301],{"type":51,"value":1302},".cpp",{"type":51,"value":441},{"type":45,"tag":60,"props":1305,"children":1307},{"className":1306},[],[1308],{"type":51,"value":1309},".hpp",{"type":51,"value":1311},"), ",{"type":45,"tag":60,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":51,"value":1317},"torch_binding.cpp",{"type":51,"value":1319},", and ",{"type":45,"tag":60,"props":1321,"children":1323},{"className":1322},[],[1324],{"type":51,"value":1325},"build.toml",{"type":51,"value":1327},". Do NOT create benchmark or test scripts.",{"type":45,"tag":1108,"props":1329,"children":1330},{},[1331,1336,1338,1344],{"type":45,"tag":76,"props":1332,"children":1333},{},[1334],{"type":51,"value":1335},"NEVER write custom timing code",{"type":51,"value":1337}," — ONLY use ",{"type":45,"tag":60,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":51,"value":1343},"scripts\u002Fbenchmark_cpu.py",{"type":51,"value":114},{"type":45,"tag":1108,"props":1346,"children":1347},{},[1348,1350,1355],{"type":51,"value":1349},"If a tool fails, ",{"type":45,"tag":76,"props":1351,"children":1352},{},[1353],{"type":51,"value":1354},"STOP and report the error",{"type":51,"value":1356},". Do NOT work around it with custom scripts.",{"type":45,"tag":1108,"props":1358,"children":1359},{},[1360,1362,1367,1369,1375,1377,1383],{"type":51,"value":1361},"Generated kernels must follow the ",{"type":45,"tag":76,"props":1363,"children":1364},{},[1365],{"type":51,"value":1366},"runtime dispatch pattern",{"type":51,"value":1368}," with ",{"type":45,"tag":60,"props":1370,"children":1372},{"className":1371},[],[1373],{"type":51,"value":1374},"cpu_features.hpp",{"type":51,"value":1376}," — see ",{"type":45,"tag":60,"props":1378,"children":1380},{"className":1379},[],[1381],{"type":51,"value":1382},"references\u002Fruntime_dispatch.yaml",{"type":51,"value":114},{"type":45,"tag":1108,"props":1385,"children":1386},{},[1387,1389,1394,1396,1402],{"type":51,"value":1388},"Every kernel should have a ",{"type":45,"tag":76,"props":1390,"children":1391},{},[1392],{"type":51,"value":1393},"generic ATen fallback",{"type":51,"value":1395}," that works on any CPU. If a specific path cannot have a meaningful fallback, use ",{"type":45,"tag":60,"props":1397,"children":1399},{"className":1398},[],[1400],{"type":51,"value":1401},"TORCH_CHECK(false, ...)",{"type":51,"value":1403}," with a clear error message.",{"type":45,"tag":1108,"props":1405,"children":1406},{},[1407,1409,1414,1416,1421,1423,1428],{"type":51,"value":1408},"Each SIMD tier (AVX2, AVX512) must be in a ",{"type":45,"tag":76,"props":1410,"children":1411},{},[1412],{"type":51,"value":1413},"separate translation unit",{"type":51,"value":1415}," (",{"type":45,"tag":60,"props":1417,"children":1419},{"className":1418},[],[1420],{"type":51,"value":1302},{"type":51,"value":1422}," file) with its own compiler flags in ",{"type":45,"tag":60,"props":1424,"children":1426},{"className":1425},[],[1427],{"type":51,"value":1325},{"type":51,"value":1429},". Do NOT mix intrinsics from different ISA levels in the same file.",{"type":45,"tag":1108,"props":1431,"children":1432},{},[1433,1435,1440],{"type":51,"value":1434},"All SIMD implementations must handle ",{"type":45,"tag":76,"props":1436,"children":1437},{},[1438],{"type":51,"value":1439},"edge cases",{"type":51,"value":1441}," (hidden_size not divisible by vector width).",{"type":45,"tag":1108,"props":1443,"children":1444},{},[1445,1447,1452],{"type":51,"value":1446},"AVX2 tier is ",{"type":45,"tag":76,"props":1448,"children":1449},{},[1450],{"type":51,"value":1451},"optional",{"type":51,"value":1453}," — most CPU kernels go directly from generic fallback to AVX512. Only add AVX2 when it provides meaningful benefit for element-wise ops.",{"type":45,"tag":1108,"props":1455,"children":1456},{},[1457,1459,1471,1473,1478],{"type":51,"value":1458},"You ",{"type":45,"tag":76,"props":1460,"children":1461},{},[1462,1464,1469],{"type":51,"value":1463},"MUST run all ",{"type":45,"tag":60,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":51,"value":112},{"type":51,"value":1470}," trials",{"type":51,"value":1472}," in Phase 2. Do NOT stop early due to plateau — the only valid early stop is speedup > ",{"type":45,"tag":60,"props":1474,"children":1476},{"className":1475},[],[1477],{"type":51,"value":104},{"type":51,"value":114},{"type":45,"tag":510,"props":1480,"children":1482},{"id":1481},"mandatory-tools",[1483],{"type":51,"value":1484},"Mandatory Tools",{"type":45,"tag":135,"props":1486,"children":1487},{},[1488,1509],{"type":45,"tag":139,"props":1489,"children":1490},{},[1491],{"type":45,"tag":143,"props":1492,"children":1493},{},[1494,1499,1504],{"type":45,"tag":147,"props":1495,"children":1496},{},[1497],{"type":51,"value":1498},"Tool",{"type":45,"tag":147,"props":1500,"children":1501},{},[1502],{"type":51,"value":1503},"Command",{"type":45,"tag":147,"props":1505,"children":1506},{},[1507],{"type":51,"value":1508},"Purpose",{"type":45,"tag":163,"props":1510,"children":1511},{},[1512,1537,1562,1587,1618,1648],{"type":45,"tag":143,"props":1513,"children":1514},{},[1515,1523,1532],{"type":45,"tag":170,"props":1516,"children":1517},{},[1518],{"type":45,"tag":76,"props":1519,"children":1520},{},[1521],{"type":51,"value":1522},"Analyze",{"type":45,"tag":170,"props":1524,"children":1525},{},[1526],{"type":45,"tag":60,"props":1527,"children":1529},{"className":1528},[],[1530],{"type":51,"value":1531},"python scripts\u002Fanalyze_op.py --op \u003Cop_name> --shapes \u003Cshapes>",{"type":45,"tag":170,"props":1533,"children":1534},{},[1535],{"type":51,"value":1536},"Analyze PyTorch op: compute\u002Fmemory characteristics, SIMD strategy recommendations",{"type":45,"tag":143,"props":1538,"children":1539},{},[1540,1548,1557],{"type":45,"tag":170,"props":1541,"children":1542},{},[1543],{"type":45,"tag":76,"props":1544,"children":1545},{},[1546],{"type":51,"value":1547},"Validate",{"type":45,"tag":170,"props":1549,"children":1550},{},[1551],{"type":45,"tag":60,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":51,"value":1556},"python scripts\u002Fvalidate_cpu_kernel.py \u003Ckernel_dir>",{"type":45,"tag":170,"props":1558,"children":1559},{},[1560],{"type":51,"value":1561},"Static checks: alignment, OpenMP usage, intrinsics correctness, build.toml validation",{"type":45,"tag":143,"props":1563,"children":1564},{},[1565,1573,1582],{"type":45,"tag":170,"props":1566,"children":1567},{},[1568],{"type":45,"tag":76,"props":1569,"children":1570},{},[1571],{"type":51,"value":1572},"Build",{"type":45,"tag":170,"props":1574,"children":1575},{},[1576],{"type":45,"tag":60,"props":1577,"children":1579},{"className":1578},[],[1580],{"type":51,"value":1581},"kernel-builder build --release",{"type":45,"tag":170,"props":1583,"children":1584},{},[1585],{"type":51,"value":1586},"Compile C++ kernel via build.toml into a wheel",{"type":45,"tag":143,"props":1588,"children":1589},{},[1590,1598,1607],{"type":45,"tag":170,"props":1591,"children":1592},{},[1593],{"type":45,"tag":76,"props":1594,"children":1595},{},[1596],{"type":51,"value":1597},"Benchmark",{"type":45,"tag":170,"props":1599,"children":1600},{},[1601],{"type":45,"tag":60,"props":1602,"children":1604},{"className":1603},[],[1605],{"type":51,"value":1606},"python scripts\u002Fbenchmark_cpu.py \u003Cbaseline_file> --kernel-package \u003Cpkg> --op \u003Cfunc>",{"type":45,"tag":170,"props":1608,"children":1609},{},[1610,1612],{"type":51,"value":1611},"Correctness + performance via ",{"type":45,"tag":60,"props":1613,"children":1615},{"className":1614},[],[1616],{"type":51,"value":1617},"torch.utils.benchmark",{"type":45,"tag":143,"props":1619,"children":1620},{},[1621,1629,1638],{"type":45,"tag":170,"props":1622,"children":1623},{},[1624],{"type":45,"tag":76,"props":1625,"children":1626},{},[1627],{"type":51,"value":1628},"Profile",{"type":45,"tag":170,"props":1630,"children":1631},{},[1632],{"type":45,"tag":60,"props":1633,"children":1635},{"className":1634},[],[1636],{"type":51,"value":1637},"python scripts\u002Fcpu_profiler.py --kernel-package \u003Cpkg> --op \u003Cfunc>",{"type":45,"tag":170,"props":1639,"children":1640},{},[1641,1646],{"type":45,"tag":60,"props":1642,"children":1644},{"className":1643},[],[1645],{"type":51,"value":1240},{"type":51,"value":1647}," hardware counters + optimization recommendations",{"type":45,"tag":143,"props":1649,"children":1650},{},[1651,1659,1668],{"type":45,"tag":170,"props":1652,"children":1653},{},[1654],{"type":45,"tag":76,"props":1655,"children":1656},{},[1657],{"type":51,"value":1658},"Trial Manager",{"type":45,"tag":170,"props":1660,"children":1661},{},[1662],{"type":45,"tag":60,"props":1663,"children":1665},{"className":1664},[],[1666],{"type":51,"value":1667},"python scripts\u002Ftrial_manager.py \u003Ccommand> ...",{"type":45,"tag":170,"props":1669,"children":1670},{},[1671],{"type":51,"value":1672},"Trial tree management (init\u002Fsave\u002Fresult\u002Fstatus\u002Fbest\u002Ffinalize)",{"type":45,"tag":69,"props":1674,"children":1675},{},[1676],{"type":45,"tag":54,"props":1677,"children":1678},{},[1679,1684,1686,1692,1693,1699],{"type":45,"tag":76,"props":1680,"children":1681},{},[1682],{"type":51,"value":1683},"Benchmark discipline",{"type":51,"value":1685},": Pin to a single NUMA node — ",{"type":45,"tag":60,"props":1687,"children":1689},{"className":1688},[],[1690],{"type":51,"value":1691},"numactl --cpunodebind=0 --membind=0 python scripts\u002Fbenchmark_cpu.py ...",{"type":51,"value":1069},{"type":45,"tag":1071,"props":1694,"children":1696},{"href":1695},"references\u002Fthreading_patterns.yaml",[1697],{"type":51,"value":1698},"threading_patterns.yaml",{"type":51,"value":114},{"type":45,"tag":510,"props":1701,"children":1703},{"id":1702},"phase-1-correctness-linear-no-branching",[1704],{"type":51,"value":1705},"Phase 1: Correctness (Linear, No Branching)",{"type":45,"tag":54,"props":1707,"children":1708},{},[1709],{"type":51,"value":1710},"Build the kernel tier by tier. Each tier must be correct before moving on.",{"type":45,"tag":1712,"props":1713,"children":1715},"h4",{"id":1714},"tier-0-generic-fallback",[1716],{"type":51,"value":1717},"Tier 0: Generic Fallback",{"type":45,"tag":1104,"props":1719,"children":1720},{},[1721,1733,1738,1743],{"type":45,"tag":1108,"props":1722,"children":1723},{},[1724,1726,1731],{"type":51,"value":1725},"Implement using ",{"type":45,"tag":76,"props":1727,"children":1728},{},[1729],{"type":51,"value":1730},"PyTorch ATen ops only",{"type":51,"value":1732}," (no intrinsics).",{"type":45,"tag":1108,"props":1734,"children":1735},{},[1736],{"type":51,"value":1737},"This serves as the portable baseline that runs on any CPU.",{"type":45,"tag":1108,"props":1739,"children":1740},{},[1741],{"type":51,"value":1742},"Must produce results matching the PyTorch reference within tolerance.",{"type":45,"tag":1108,"props":1744,"children":1745},{},[1746,1748],{"type":51,"value":1747},"File: ",{"type":45,"tag":60,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":51,"value":1753},"\u003Ckernel>_cpu\u002F\u003Ckernel>_cpu.cpp",{"type":45,"tag":543,"props":1755,"children":1757},{"className":545,"code":1756,"language":547,"meta":548,"style":548},"# Validate + build\npython scripts\u002Fvalidate_cpu_kernel.py .\nkernel-builder build --release\npip install dist\u002F*.whl --force-reinstall\n\n# Benchmark (this also establishes the PyTorch baseline time)\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package my_kernel --op my_kernel.rms_norm\n",[1758],{"type":45,"tag":60,"props":1759,"children":1760},{"__ignoreMap":548},[1761,1769,1786,1802,1830,1837,1845],{"type":45,"tag":554,"props":1762,"children":1763},{"class":556,"line":557},[1764],{"type":45,"tag":554,"props":1765,"children":1766},{"style":561},[1767],{"type":51,"value":1768},"# Validate + build\n",{"type":45,"tag":554,"props":1770,"children":1771},{"class":556,"line":567},[1772,1776,1781],{"type":45,"tag":554,"props":1773,"children":1774},{"style":580},[1775],{"type":51,"value":583},{"type":45,"tag":554,"props":1777,"children":1778},{"style":586},[1779],{"type":51,"value":1780}," scripts\u002Fvalidate_cpu_kernel.py",{"type":45,"tag":554,"props":1782,"children":1783},{"style":586},[1784],{"type":51,"value":1785}," .\n",{"type":45,"tag":554,"props":1787,"children":1788},{"class":556,"line":576},[1789,1793,1797],{"type":45,"tag":554,"props":1790,"children":1791},{"style":580},[1792],{"type":51,"value":65},{"type":45,"tag":554,"props":1794,"children":1795},{"style":586},[1796],{"type":51,"value":731},{"type":45,"tag":554,"props":1798,"children":1799},{"style":586},[1800],{"type":51,"value":1801}," --release\n",{"type":45,"tag":554,"props":1803,"children":1804},{"class":556,"line":632},[1805,1810,1814,1818,1822,1826],{"type":45,"tag":554,"props":1806,"children":1807},{"style":580},[1808],{"type":51,"value":1809},"pip",{"type":45,"tag":554,"props":1811,"children":1812},{"style":586},[1813],{"type":51,"value":750},{"type":45,"tag":554,"props":1815,"children":1816},{"style":586},[1817],{"type":51,"value":755},{"type":45,"tag":554,"props":1819,"children":1820},{"style":758},[1821],{"type":51,"value":761},{"type":45,"tag":554,"props":1823,"children":1824},{"style":586},[1825],{"type":51,"value":766},{"type":45,"tag":554,"props":1827,"children":1828},{"style":586},[1829],{"type":51,"value":771},{"type":45,"tag":554,"props":1831,"children":1832},{"class":556,"line":641},[1833],{"type":45,"tag":554,"props":1834,"children":1835},{"emptyLinePlaceholder":38},[1836],{"type":51,"value":638},{"type":45,"tag":554,"props":1838,"children":1839},{"class":556,"line":650},[1840],{"type":45,"tag":554,"props":1841,"children":1842},{"style":561},[1843],{"type":51,"value":1844},"# Benchmark (this also establishes the PyTorch baseline time)\n",{"type":45,"tag":554,"props":1846,"children":1847},{"class":556,"line":659},[1848,1852,1856,1860,1864,1868,1872],{"type":45,"tag":554,"props":1849,"children":1850},{"style":580},[1851],{"type":51,"value":583},{"type":45,"tag":554,"props":1853,"children":1854},{"style":586},[1855],{"type":51,"value":810},{"type":45,"tag":554,"props":1857,"children":1858},{"style":586},[1859],{"type":51,"value":815},{"type":45,"tag":554,"props":1861,"children":1862},{"style":586},[1863],{"type":51,"value":820},{"type":45,"tag":554,"props":1865,"children":1866},{"style":586},[1867],{"type":51,"value":825},{"type":45,"tag":554,"props":1869,"children":1870},{"style":586},[1871],{"type":51,"value":594},{"type":45,"tag":554,"props":1873,"children":1874},{"style":586},[1875],{"type":51,"value":834},{"type":45,"tag":1712,"props":1877,"children":1879},{"id":1878},"tier-1-avx2-optional",[1880],{"type":51,"value":1881},"Tier 1: AVX2 (Optional)",{"type":45,"tag":1104,"props":1883,"children":1884},{},[1885,1898,1910,1915],{"type":45,"tag":1108,"props":1886,"children":1887},{},[1888,1890,1896],{"type":51,"value":1889},"Add AVX2 implementation using ",{"type":45,"tag":60,"props":1891,"children":1893},{"className":1892},[],[1894],{"type":51,"value":1895},"_mm256_*",{"type":51,"value":1897}," intrinsics.",{"type":45,"tag":1108,"props":1899,"children":1900},{},[1901,1903,1909],{"type":51,"value":1902},"Compile with ",{"type":45,"tag":60,"props":1904,"children":1906},{"className":1905},[],[1907],{"type":51,"value":1908},"-mavx2 -mfma -mf16c -fopenmp",{"type":51,"value":114},{"type":45,"tag":1108,"props":1911,"children":1912},{},[1913],{"type":51,"value":1914},"Must be correct; performance improvement is a bonus.",{"type":45,"tag":1108,"props":1916,"children":1917},{},[1918,1919],{"type":51,"value":1747},{"type":45,"tag":60,"props":1920,"children":1922},{"className":1921},[],[1923],{"type":51,"value":1924},"\u003Ckernel>_cpu\u002F\u003Ckernel>_avx2.cpp",{"type":45,"tag":1712,"props":1926,"children":1928},{"id":1927},"tier-2-avx512",[1929],{"type":51,"value":1930},"Tier 2: AVX512",{"type":45,"tag":1104,"props":1932,"children":1933},{},[1934,1946,1957,1969,1981],{"type":45,"tag":1108,"props":1935,"children":1936},{},[1937,1939,1945],{"type":51,"value":1938},"Add AVX512 implementation using ",{"type":45,"tag":60,"props":1940,"children":1942},{"className":1941},[],[1943],{"type":51,"value":1944},"_mm512_*",{"type":51,"value":1897},{"type":45,"tag":1108,"props":1947,"children":1948},{},[1949,1950,1956],{"type":51,"value":1902},{"type":45,"tag":60,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":51,"value":1955},"-mavx512f -mavx512bf16 -mavx512vl -mavx512dq -mavx512bw -mavx512vbmi -mfma -mf16c -fopenmp",{"type":51,"value":114},{"type":45,"tag":1108,"props":1958,"children":1959},{},[1960,1962,1968],{"type":51,"value":1961},"For GEMM kernels using brgemm, additionally add ",{"type":45,"tag":60,"props":1963,"children":1965},{"className":1964},[],[1966],{"type":51,"value":1967},"-mamx-tile -mamx-bf16 -mamx-int8",{"type":51,"value":114},{"type":45,"tag":1108,"props":1970,"children":1971},{},[1972,1974,1979],{"type":51,"value":1973},"This is the ",{"type":45,"tag":76,"props":1975,"children":1976},{},[1977],{"type":51,"value":1978},"entry point to Phase 2",{"type":51,"value":1980}," — performance optimization starts here.",{"type":45,"tag":1108,"props":1982,"children":1983},{},[1984,1985],{"type":51,"value":1747},{"type":45,"tag":60,"props":1986,"children":1988},{"className":1987},[],[1989],{"type":51,"value":1990},"\u003Ckernel>_cpu\u002F\u003Ckernel>_avx512.cpp",{"type":45,"tag":510,"props":1992,"children":1994},{"id":1993},"phase-2-performance-exploration-branching-with-backtracking",[1995],{"type":51,"value":1996},"Phase 2: Performance Exploration (Branching, With Backtracking)",{"type":45,"tag":54,"props":1998,"children":1999},{},[2000],{"type":51,"value":2001},"Once AVX512 is correct, optimize for peak performance. This phase uses the trial manager with branching.",{"type":45,"tag":543,"props":2003,"children":2005},{"className":545,"code":2004,"language":547,"meta":548,"style":548},"# Initialize Phase 2 trials\npython scripts\u002Ftrial_manager.py init \u003Ckernel_name> baseline.py\n\n# For each trial:\n# 1. Modify kernel code (AVX512 intrinsics, or brgemm for GEMM kernels)\n# 2. Build\nkernel-builder build --release && pip install dist\u002F*.whl --force-reinstall\n# 3. Benchmark\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n# 4. Save trial\npython scripts\u002Ftrial_manager.py save \u003Ckernel_name> \u003Cdir> --parent \u003Cparent_id> --strategy \"description\"\n# 5. Record result\npython scripts\u002Ftrial_manager.py result \u003Ckernel_name> \u003Ctrial_id> --correctness pass --speedup \u003Cfloat> --baseline_us \u003Cfloat> --kernel_us \u003Cfloat>\n# 6. Profile (after t1, or when plateaued)\npython scripts\u002Fcpu_profiler.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n",[2006],{"type":45,"tag":60,"props":2007,"children":2008},{"__ignoreMap":548},[2009,2017,2056,2063,2071,2079,2087,2130,2138,2197,2205,2296,2304,2428,2436],{"type":45,"tag":554,"props":2010,"children":2011},{"class":556,"line":557},[2012],{"type":45,"tag":554,"props":2013,"children":2014},{"style":561},[2015],{"type":51,"value":2016},"# Initialize Phase 2 trials\n",{"type":45,"tag":554,"props":2018,"children":2019},{"class":556,"line":567},[2020,2024,2028,2032,2037,2042,2047,2052],{"type":45,"tag":554,"props":2021,"children":2022},{"style":580},[2023],{"type":51,"value":583},{"type":45,"tag":554,"props":2025,"children":2026},{"style":586},[2027],{"type":51,"value":669},{"type":45,"tag":554,"props":2029,"children":2030},{"style":586},[2031],{"type":51,"value":674},{"type":45,"tag":554,"props":2033,"children":2034},{"style":597},[2035],{"type":51,"value":2036}," \u003C",{"type":45,"tag":554,"props":2038,"children":2039},{"style":586},[2040],{"type":51,"value":2041},"kernel_nam",{"type":45,"tag":554,"props":2043,"children":2044},{"style":758},[2045],{"type":51,"value":2046},"e",{"type":45,"tag":554,"props":2048,"children":2049},{"style":597},[2050],{"type":51,"value":2051},">",{"type":45,"tag":554,"props":2053,"children":2054},{"style":586},[2055],{"type":51,"value":684},{"type":45,"tag":554,"props":2057,"children":2058},{"class":556,"line":576},[2059],{"type":45,"tag":554,"props":2060,"children":2061},{"emptyLinePlaceholder":38},[2062],{"type":51,"value":638},{"type":45,"tag":554,"props":2064,"children":2065},{"class":556,"line":632},[2066],{"type":45,"tag":554,"props":2067,"children":2068},{"style":561},[2069],{"type":51,"value":2070},"# For each trial:\n",{"type":45,"tag":554,"props":2072,"children":2073},{"class":556,"line":641},[2074],{"type":45,"tag":554,"props":2075,"children":2076},{"style":561},[2077],{"type":51,"value":2078},"# 1. Modify kernel code (AVX512 intrinsics, or brgemm for GEMM kernels)\n",{"type":45,"tag":554,"props":2080,"children":2081},{"class":556,"line":650},[2082],{"type":45,"tag":554,"props":2083,"children":2084},{"style":561},[2085],{"type":51,"value":2086},"# 2. Build\n",{"type":45,"tag":554,"props":2088,"children":2089},{"class":556,"line":659},[2090,2094,2098,2102,2106,2110,2114,2118,2122,2126],{"type":45,"tag":554,"props":2091,"children":2092},{"style":580},[2093],{"type":51,"value":65},{"type":45,"tag":554,"props":2095,"children":2096},{"style":586},[2097],{"type":51,"value":731},{"type":45,"tag":554,"props":2099,"children":2100},{"style":586},[2101],{"type":51,"value":736},{"type":45,"tag":554,"props":2103,"children":2104},{"style":597},[2105],{"type":51,"value":721},{"type":45,"tag":554,"props":2107,"children":2108},{"style":580},[2109],{"type":51,"value":745},{"type":45,"tag":554,"props":2111,"children":2112},{"style":586},[2113],{"type":51,"value":750},{"type":45,"tag":554,"props":2115,"children":2116},{"style":586},[2117],{"type":51,"value":755},{"type":45,"tag":554,"props":2119,"children":2120},{"style":758},[2121],{"type":51,"value":761},{"type":45,"tag":554,"props":2123,"children":2124},{"style":586},[2125],{"type":51,"value":766},{"type":45,"tag":554,"props":2127,"children":2128},{"style":586},[2129],{"type":51,"value":771},{"type":45,"tag":554,"props":2131,"children":2132},{"class":556,"line":687},[2133],{"type":45,"tag":554,"props":2134,"children":2135},{"style":561},[2136],{"type":51,"value":2137},"# 3. Benchmark\n",{"type":45,"tag":554,"props":2139,"children":2140},{"class":556,"line":695},[2141,2145,2149,2153,2157,2161,2166,2171,2175,2179,2183,2188,2192],{"type":45,"tag":554,"props":2142,"children":2143},{"style":580},[2144],{"type":51,"value":583},{"type":45,"tag":554,"props":2146,"children":2147},{"style":586},[2148],{"type":51,"value":810},{"type":45,"tag":554,"props":2150,"children":2151},{"style":586},[2152],{"type":51,"value":815},{"type":45,"tag":554,"props":2154,"children":2155},{"style":586},[2156],{"type":51,"value":820},{"type":45,"tag":554,"props":2158,"children":2159},{"style":597},[2160],{"type":51,"value":2036},{"type":45,"tag":554,"props":2162,"children":2163},{"style":586},[2164],{"type":51,"value":2165},"pk",{"type":45,"tag":554,"props":2167,"children":2168},{"style":758},[2169],{"type":51,"value":2170},"g",{"type":45,"tag":554,"props":2172,"children":2173},{"style":597},[2174],{"type":51,"value":2051},{"type":45,"tag":554,"props":2176,"children":2177},{"style":586},[2178],{"type":51,"value":594},{"type":45,"tag":554,"props":2180,"children":2181},{"style":597},[2182],{"type":51,"value":2036},{"type":45,"tag":554,"props":2184,"children":2185},{"style":586},[2186],{"type":51,"value":2187},"fun",{"type":45,"tag":554,"props":2189,"children":2190},{"style":758},[2191],{"type":51,"value":20},{"type":45,"tag":554,"props":2193,"children":2194},{"style":597},[2195],{"type":51,"value":2196},">\n",{"type":45,"tag":554,"props":2198,"children":2199},{"class":556,"line":704},[2200],{"type":45,"tag":554,"props":2201,"children":2202},{"style":561},[2203],{"type":51,"value":2204},"# 4. Save trial\n",{"type":45,"tag":554,"props":2206,"children":2207},{"class":556,"line":774},[2208,2212,2216,2221,2225,2229,2233,2237,2241,2246,2251,2255,2260,2264,2269,2274,2278,2283,2287,2292],{"type":45,"tag":554,"props":2209,"children":2210},{"style":580},[2211],{"type":51,"value":583},{"type":45,"tag":554,"props":2213,"children":2214},{"style":586},[2215],{"type":51,"value":669},{"type":45,"tag":554,"props":2217,"children":2218},{"style":586},[2219],{"type":51,"value":2220}," save",{"type":45,"tag":554,"props":2222,"children":2223},{"style":597},[2224],{"type":51,"value":2036},{"type":45,"tag":554,"props":2226,"children":2227},{"style":586},[2228],{"type":51,"value":2041},{"type":45,"tag":554,"props":2230,"children":2231},{"style":758},[2232],{"type":51,"value":2046},{"type":45,"tag":554,"props":2234,"children":2235},{"style":597},[2236],{"type":51,"value":2051},{"type":45,"tag":554,"props":2238,"children":2239},{"style":597},[2240],{"type":51,"value":2036},{"type":45,"tag":554,"props":2242,"children":2243},{"style":586},[2244],{"type":51,"value":2245},"di",{"type":45,"tag":554,"props":2247,"children":2248},{"style":758},[2249],{"type":51,"value":2250},"r",{"type":45,"tag":554,"props":2252,"children":2253},{"style":597},[2254],{"type":51,"value":2051},{"type":45,"tag":554,"props":2256,"children":2257},{"style":586},[2258],{"type":51,"value":2259}," --parent",{"type":45,"tag":554,"props":2261,"children":2262},{"style":597},[2263],{"type":51,"value":2036},{"type":45,"tag":554,"props":2265,"children":2266},{"style":586},[2267],{"type":51,"value":2268},"parent_i",{"type":45,"tag":554,"props":2270,"children":2271},{"style":758},[2272],{"type":51,"value":2273},"d",{"type":45,"tag":554,"props":2275,"children":2276},{"style":597},[2277],{"type":51,"value":2051},{"type":45,"tag":554,"props":2279,"children":2280},{"style":586},[2281],{"type":51,"value":2282}," --strategy",{"type":45,"tag":554,"props":2284,"children":2285},{"style":597},[2286],{"type":51,"value":600},{"type":45,"tag":554,"props":2288,"children":2289},{"style":586},[2290],{"type":51,"value":2291},"description",{"type":45,"tag":554,"props":2293,"children":2294},{"style":597},[2295],{"type":51,"value":629},{"type":45,"tag":554,"props":2297,"children":2298},{"class":556,"line":782},[2299],{"type":45,"tag":554,"props":2300,"children":2301},{"style":561},[2302],{"type":51,"value":2303},"# 5. Record result\n",{"type":45,"tag":554,"props":2305,"children":2306},{"class":556,"line":791},[2307,2311,2315,2320,2324,2328,2332,2336,2340,2345,2349,2353,2358,2363,2368,2372,2377,2382,2386,2391,2395,2399,2403,2407,2412,2416,2420,2424],{"type":45,"tag":554,"props":2308,"children":2309},{"style":580},[2310],{"type":51,"value":583},{"type":45,"tag":554,"props":2312,"children":2313},{"style":586},[2314],{"type":51,"value":669},{"type":45,"tag":554,"props":2316,"children":2317},{"style":586},[2318],{"type":51,"value":2319}," result",{"type":45,"tag":554,"props":2321,"children":2322},{"style":597},[2323],{"type":51,"value":2036},{"type":45,"tag":554,"props":2325,"children":2326},{"style":586},[2327],{"type":51,"value":2041},{"type":45,"tag":554,"props":2329,"children":2330},{"style":758},[2331],{"type":51,"value":2046},{"type":45,"tag":554,"props":2333,"children":2334},{"style":597},[2335],{"type":51,"value":2051},{"type":45,"tag":554,"props":2337,"children":2338},{"style":597},[2339],{"type":51,"value":2036},{"type":45,"tag":554,"props":2341,"children":2342},{"style":586},[2343],{"type":51,"value":2344},"trial_i",{"type":45,"tag":554,"props":2346,"children":2347},{"style":758},[2348],{"type":51,"value":2273},{"type":45,"tag":554,"props":2350,"children":2351},{"style":597},[2352],{"type":51,"value":2051},{"type":45,"tag":554,"props":2354,"children":2355},{"style":586},[2356],{"type":51,"value":2357}," --correctness",{"type":45,"tag":554,"props":2359,"children":2360},{"style":586},[2361],{"type":51,"value":2362}," pass",{"type":45,"tag":554,"props":2364,"children":2365},{"style":586},[2366],{"type":51,"value":2367}," --speedup",{"type":45,"tag":554,"props":2369,"children":2370},{"style":597},[2371],{"type":51,"value":2036},{"type":45,"tag":554,"props":2373,"children":2374},{"style":586},[2375],{"type":51,"value":2376},"floa",{"type":45,"tag":554,"props":2378,"children":2379},{"style":758},[2380],{"type":51,"value":2381},"t",{"type":45,"tag":554,"props":2383,"children":2384},{"style":597},[2385],{"type":51,"value":2051},{"type":45,"tag":554,"props":2387,"children":2388},{"style":586},[2389],{"type":51,"value":2390}," --baseline_us",{"type":45,"tag":554,"props":2392,"children":2393},{"style":597},[2394],{"type":51,"value":2036},{"type":45,"tag":554,"props":2396,"children":2397},{"style":586},[2398],{"type":51,"value":2376},{"type":45,"tag":554,"props":2400,"children":2401},{"style":758},[2402],{"type":51,"value":2381},{"type":45,"tag":554,"props":2404,"children":2405},{"style":597},[2406],{"type":51,"value":2051},{"type":45,"tag":554,"props":2408,"children":2409},{"style":586},[2410],{"type":51,"value":2411}," --kernel_us",{"type":45,"tag":554,"props":2413,"children":2414},{"style":597},[2415],{"type":51,"value":2036},{"type":45,"tag":554,"props":2417,"children":2418},{"style":586},[2419],{"type":51,"value":2376},{"type":45,"tag":554,"props":2421,"children":2422},{"style":758},[2423],{"type":51,"value":2381},{"type":45,"tag":554,"props":2425,"children":2426},{"style":597},[2427],{"type":51,"value":2196},{"type":45,"tag":554,"props":2429,"children":2430},{"class":556,"line":800},[2431],{"type":45,"tag":554,"props":2432,"children":2433},{"style":561},[2434],{"type":51,"value":2435},"# 6. Profile (after t1, or when plateaued)\n",{"type":45,"tag":554,"props":2437,"children":2438},{"class":556,"line":837},[2439,2443,2447,2451,2455,2459,2463,2467,2471,2475,2479,2483],{"type":45,"tag":554,"props":2440,"children":2441},{"style":580},[2442],{"type":51,"value":583},{"type":45,"tag":554,"props":2444,"children":2445},{"style":586},[2446],{"type":51,"value":864},{"type":45,"tag":554,"props":2448,"children":2449},{"style":586},[2450],{"type":51,"value":820},{"type":45,"tag":554,"props":2452,"children":2453},{"style":597},[2454],{"type":51,"value":2036},{"type":45,"tag":554,"props":2456,"children":2457},{"style":586},[2458],{"type":51,"value":2165},{"type":45,"tag":554,"props":2460,"children":2461},{"style":758},[2462],{"type":51,"value":2170},{"type":45,"tag":554,"props":2464,"children":2465},{"style":597},[2466],{"type":51,"value":2051},{"type":45,"tag":554,"props":2468,"children":2469},{"style":586},[2470],{"type":51,"value":594},{"type":45,"tag":554,"props":2472,"children":2473},{"style":597},[2474],{"type":51,"value":2036},{"type":45,"tag":554,"props":2476,"children":2477},{"style":586},[2478],{"type":51,"value":2187},{"type":45,"tag":554,"props":2480,"children":2481},{"style":758},[2482],{"type":51,"value":20},{"type":45,"tag":554,"props":2484,"children":2485},{"style":597},[2486],{"type":51,"value":2196},{"type":45,"tag":1712,"props":2488,"children":2490},{"id":2489},"phase-2-decision-tree",[2491],{"type":51,"value":2492},"Phase 2 Decision Tree",{"type":45,"tag":135,"props":2494,"children":2495},{},[2496,2512],{"type":45,"tag":139,"props":2497,"children":2498},{},[2499],{"type":45,"tag":143,"props":2500,"children":2501},{},[2502,2507],{"type":45,"tag":147,"props":2503,"children":2504},{},[2505],{"type":51,"value":2506},"Condition",{"type":45,"tag":147,"props":2508,"children":2509},{},[2510],{"type":51,"value":2511},"Action",{"type":45,"tag":163,"props":2513,"children":2514},{},[2515,2536,2552,2568,2584,2614,2630,2646,2662,2685],{"type":45,"tag":143,"props":2516,"children":2517},{},[2518,2531],{"type":45,"tag":170,"props":2519,"children":2520},{},[2521],{"type":45,"tag":76,"props":2522,"children":2523},{},[2524,2526],{"type":51,"value":2525},"Speedup > ",{"type":45,"tag":60,"props":2527,"children":2529},{"className":2528},[],[2530],{"type":51,"value":104},{"type":45,"tag":170,"props":2532,"children":2533},{},[2534],{"type":51,"value":2535},"Stop — excellent result (the only valid early stop)",{"type":45,"tag":143,"props":2537,"children":2538},{},[2539,2547],{"type":45,"tag":170,"props":2540,"children":2541},{},[2542],{"type":45,"tag":76,"props":2543,"children":2544},{},[2545],{"type":51,"value":2546},"Speedup improved",{"type":45,"tag":170,"props":2548,"children":2549},{},[2550],{"type":51,"value":2551},"Continue on this branch, try next optimization",{"type":45,"tag":143,"props":2553,"children":2554},{},[2555,2563],{"type":45,"tag":170,"props":2556,"children":2557},{},[2558],{"type":45,"tag":76,"props":2559,"children":2560},{},[2561],{"type":51,"value":2562},"Speedup regressed",{"type":45,"tag":170,"props":2564,"children":2565},{},[2566],{"type":51,"value":2567},"Branch back to best trial, try different strategy",{"type":45,"tag":143,"props":2569,"children":2570},{},[2571,2579],{"type":45,"tag":170,"props":2572,"children":2573},{},[2574],{"type":45,"tag":76,"props":2575,"children":2576},{},[2577],{"type":51,"value":2578},"Correctness failed",{"type":45,"tag":170,"props":2580,"children":2581},{},[2582],{"type":51,"value":2583},"Fix on same branch (usually alignment or SIMD boundary bug)",{"type":45,"tag":143,"props":2585,"children":2586},{},[2587,2602],{"type":45,"tag":170,"props":2588,"children":2589},{},[2590],{"type":45,"tag":76,"props":2591,"children":2592},{},[2593,2595,2600],{"type":51,"value":2594},"After t1 (if ",{"type":45,"tag":60,"props":2596,"children":2598},{"className":2597},[],[2599],{"type":51,"value":1224},{"type":51,"value":2601},")",{"type":45,"tag":170,"props":2603,"children":2604},{},[2605,2607,2612],{"type":51,"value":2606},"Run ",{"type":45,"tag":60,"props":2608,"children":2610},{"className":2609},[],[2611],{"type":51,"value":476},{"type":51,"value":2613}," — mandatory first profile",{"type":45,"tag":143,"props":2615,"children":2616},{},[2617,2625],{"type":45,"tag":170,"props":2618,"children":2619},{},[2620],{"type":45,"tag":76,"props":2621,"children":2622},{},[2623],{"type":51,"value":2624},"IPC \u003C 1.0",{"type":45,"tag":170,"props":2626,"children":2627},{},[2628],{"type":51,"value":2629},"Memory bound → add prefetch, change cache blocking",{"type":45,"tag":143,"props":2631,"children":2632},{},[2633,2641],{"type":45,"tag":170,"props":2634,"children":2635},{},[2636],{"type":45,"tag":76,"props":2637,"children":2638},{},[2639],{"type":51,"value":2640},"L1 miss rate high",{"type":45,"tag":170,"props":2642,"children":2643},{},[2644],{"type":51,"value":2645},"Tile too large for L1 → reduce tile size",{"type":45,"tag":143,"props":2647,"children":2648},{},[2649,2657],{"type":45,"tag":170,"props":2650,"children":2651},{},[2652],{"type":45,"tag":76,"props":2653,"children":2654},{},[2655],{"type":51,"value":2656},"L3 miss rate high",{"type":45,"tag":170,"props":2658,"children":2659},{},[2660],{"type":51,"value":2661},"Working set too large → add cache blocking",{"type":45,"tag":143,"props":2663,"children":2664},{},[2665,2673],{"type":45,"tag":170,"props":2666,"children":2667},{},[2668],{"type":45,"tag":76,"props":2669,"children":2670},{},[2671],{"type":51,"value":2672},"Plateau after 2+ trials",{"type":45,"tag":170,"props":2674,"children":2675},{},[2676,2678,2683],{"type":51,"value":2677},"Do NOT keep tuning the same knobs. Change the ",{"type":45,"tag":76,"props":2679,"children":2680},{},[2681],{"type":51,"value":2682},"approach",{"type":51,"value":2684},": switch algorithm path (tinygemm ↔ brgemm), change the fusion\u002Fblocking\u002Fdata-layout strategy, or reconsider the dispatch heuristic. A different structure beats endless parameter sweeps.",{"type":45,"tag":143,"props":2686,"children":2687},{},[2688,2696],{"type":45,"tag":170,"props":2689,"children":2690},{},[2691],{"type":45,"tag":76,"props":2692,"children":2693},{},[2694],{"type":51,"value":2695},"Max trials reached",{"type":45,"tag":170,"props":2697,"children":2698},{},[2699,2701,2706,2708],{"type":51,"value":2700},"Stop — must run all ",{"type":45,"tag":60,"props":2702,"children":2704},{"className":2703},[],[2705],{"type":51,"value":112},{"type":51,"value":2707}," from ",{"type":45,"tag":60,"props":2709,"children":2711},{"className":2710},[],[2712],{"type":51,"value":1168},{"type":45,"tag":1712,"props":2714,"children":2716},{"id":2715},"optimization-search-space-phase-2",[2717],{"type":51,"value":2718},"Optimization Search Space (Phase 2)",{"type":45,"tag":69,"props":2720,"children":2721},{},[2722],{"type":45,"tag":54,"props":2723,"children":2724},{},[2725,2727,2732,2734,2740],{"type":51,"value":2726},"These tables are a ",{"type":45,"tag":76,"props":2728,"children":2729},{},[2730],{"type":51,"value":2731},"starting menu of values seen in existing kernels, not an exhaustive recipe",{"type":51,"value":2733},". Use them to seed trials, but when a branch plateaus, prefer a structurally different idea (algorithm, fusion, memory strategy) over sweeping these knobs further. See the try-harder tree in ",{"type":45,"tag":1071,"props":2735,"children":2737},{"href":2736},"references\u002Foptimization_levels.yaml",[2738],{"type":51,"value":2739},"optimization_levels.yaml",{"type":51,"value":114},{"type":45,"tag":54,"props":2742,"children":2743},{},[2744],{"type":45,"tag":76,"props":2745,"children":2746},{},[2747],{"type":51,"value":2748},"GEMM kernels (quantized GEMM, Flash Attention, MoE):",{"type":45,"tag":135,"props":2750,"children":2751},{},[2752,2773],{"type":45,"tag":139,"props":2753,"children":2754},{},[2755],{"type":45,"tag":143,"props":2756,"children":2757},{},[2758,2763,2768],{"type":45,"tag":147,"props":2759,"children":2760},{},[2761],{"type":51,"value":2762},"Dimension",{"type":45,"tag":147,"props":2764,"children":2765},{},[2766],{"type":51,"value":2767},"Actual Values in Existing Kernels",{"type":45,"tag":147,"props":2769,"children":2770},{},[2771],{"type":51,"value":2772},"Notes",{"type":45,"tag":163,"props":2774,"children":2775},{},[2776,2794,2812,2830,2848,2866,2884,2909,2933,2957,2981,3005],{"type":45,"tag":143,"props":2777,"children":2778},{},[2779,2784,2789],{"type":45,"tag":170,"props":2780,"children":2781},{},[2782],{"type":51,"value":2783},"BLOCK_M (tinygemm path)",{"type":45,"tag":170,"props":2785,"children":2786},{},[2787],{"type":51,"value":2788},"4",{"type":45,"tag":170,"props":2790,"children":2791},{},[2792],{"type":51,"value":2793},"Small M, fused dequant+GEMM",{"type":45,"tag":143,"props":2795,"children":2796},{},[2797,2802,2807],{"type":45,"tag":170,"props":2798,"children":2799},{},[2800],{"type":51,"value":2801},"BLOCK_M (brgemm path)",{"type":45,"tag":170,"props":2803,"children":2804},{},[2805],{"type":51,"value":2806},"32 (= 2×TILE_M)",{"type":45,"tag":170,"props":2808,"children":2809},{},[2810],{"type":51,"value":2811},"Must be multiple of TILE_M=16",{"type":45,"tag":143,"props":2813,"children":2814},{},[2815,2820,2825],{"type":45,"tag":170,"props":2816,"children":2817},{},[2818],{"type":51,"value":2819},"BLOCK_N (tinygemm)",{"type":45,"tag":170,"props":2821,"children":2822},{},[2823],{"type":51,"value":2824},"32, 64",{"type":45,"tag":170,"props":2826,"children":2827},{},[2828],{"type":51,"value":2829},"Determines register tile COLS = BLOCK_N\u002F16",{"type":45,"tag":143,"props":2831,"children":2832},{},[2833,2838,2843],{"type":45,"tag":170,"props":2834,"children":2835},{},[2836],{"type":51,"value":2837},"BLOCK_N (brgemm)",{"type":45,"tag":170,"props":2839,"children":2840},{},[2841],{"type":51,"value":2842},"32 (= 2×TILE_N)",{"type":45,"tag":170,"props":2844,"children":2845},{},[2846],{"type":51,"value":2847},"Must be multiple of TILE_N=16",{"type":45,"tag":143,"props":2849,"children":2850},{},[2851,2856,2861],{"type":45,"tag":170,"props":2852,"children":2853},{},[2854],{"type":51,"value":2855},"BLOCK_K",{"type":45,"tag":170,"props":2857,"children":2858},{},[2859],{"type":51,"value":2860},"128 (= 4×TILE_K)",{"type":45,"tag":170,"props":2862,"children":2863},{},[2864],{"type":51,"value":2865},"K-dimension blocking",{"type":45,"tag":143,"props":2867,"children":2868},{},[2869,2874,2879],{"type":45,"tag":170,"props":2870,"children":2871},{},[2872],{"type":51,"value":2873},"BLOCK_M\u002FN (flash-attn2)",{"type":45,"tag":170,"props":2875,"children":2876},{},[2877],{"type":51,"value":2878},"256 \u002F 768",{"type":45,"tag":170,"props":2880,"children":2881},{},[2882],{"type":51,"value":2883},"Much larger — attention-specific",{"type":45,"tag":143,"props":2885,"children":2886},{},[2887,2892,2904],{"type":45,"tag":170,"props":2888,"children":2889},{},[2890],{"type":51,"value":2891},"K-loop unroll",{"type":45,"tag":170,"props":2893,"children":2894},{},[2895,2897,2903],{"type":51,"value":2896},"4 (",{"type":45,"tag":60,"props":2898,"children":2900},{"className":2899},[],[2901],{"type":51,"value":2902},"#pragma GCC unroll 4",{"type":51,"value":2601},{"type":45,"tag":170,"props":2905,"children":2906},{},[2907],{"type":51,"value":2908},"All GEMM kernels use 4",{"type":45,"tag":143,"props":2910,"children":2911},{},[2912,2917,2922],{"type":45,"tag":170,"props":2913,"children":2914},{},[2915],{"type":51,"value":2916},"Prefetch distance",{"type":45,"tag":170,"props":2918,"children":2919},{},[2920],{"type":51,"value":2921},"0 (disabled), 64 elements ahead",{"type":45,"tag":170,"props":2923,"children":2924},{},[2925,2927],{"type":51,"value":2926},"L1 prefetch via ",{"type":45,"tag":60,"props":2928,"children":2930},{"className":2929},[],[2931],{"type":51,"value":2932},"_MM_HINT_T0",{"type":45,"tag":143,"props":2934,"children":2935},{},[2936,2941,2952],{"type":45,"tag":170,"props":2937,"children":2938},{},[2939],{"type":51,"value":2940},"Algorithm path",{"type":45,"tag":170,"props":2942,"children":2943},{},[2944,2950],{"type":45,"tag":60,"props":2945,"children":2947},{"className":2946},[],[2948],{"type":51,"value":2949},"use_brgemm",{"type":51,"value":2951}," threshold (e.g. M > 4)",{"type":45,"tag":170,"props":2953,"children":2954},{},[2955],{"type":51,"value":2956},"Switch from tinygemm to brgemm",{"type":45,"tag":143,"props":2958,"children":2959},{},[2960,2965,2976],{"type":45,"tag":170,"props":2961,"children":2962},{},[2963],{"type":51,"value":2964},"brgemm dequant policy",{"type":45,"tag":170,"props":2966,"children":2967},{},[2968,2974],{"type":45,"tag":60,"props":2969,"children":2971},{"className":2970},[],[2972],{"type":51,"value":2973},"use_brgemm_dequant_out",{"type":51,"value":2975}," (e.g. M > 100)",{"type":45,"tag":170,"props":2977,"children":2978},{},[2979],{"type":51,"value":2980},"True=pre-dequant all B upfront; False=dequant per K-block",{"type":45,"tag":143,"props":2982,"children":2983},{},[2984,2989,2994],{"type":45,"tag":170,"props":2985,"children":2986},{},[2987],{"type":51,"value":2988},"L2 cache budget",{"type":45,"tag":170,"props":2990,"children":2991},{},[2992],{"type":51,"value":2993},"1 MB (50% of 2 MB L2)",{"type":45,"tag":170,"props":2995,"children":2996},{},[2997,2999],{"type":51,"value":2998},"Controls N-blocking in ",{"type":45,"tag":60,"props":3000,"children":3002},{"className":3001},[],[3003],{"type":51,"value":3004},"loop_2d",{"type":45,"tag":143,"props":3006,"children":3007},{},[3008,3013,3018],{"type":45,"tag":170,"props":3009,"children":3010},{},[3011],{"type":51,"value":3012},"Thread decomposition",{"type":45,"tag":170,"props":3014,"children":3015},{},[3016],{"type":51,"value":3017},"2D factorization: nth_m × nth_n",{"type":45,"tag":170,"props":3019,"children":3020},{},[3021],{"type":51,"value":3022},"Based on M\u002FN aspect ratio",{"type":45,"tag":54,"props":3024,"children":3025},{},[3026],{"type":45,"tag":76,"props":3027,"children":3028},{},[3029],{"type":51,"value":3030},"Element-wise kernels (RMSNorm, activations):",{"type":45,"tag":135,"props":3032,"children":3033},{},[3034,3053],{"type":45,"tag":139,"props":3035,"children":3036},{},[3037],{"type":45,"tag":143,"props":3038,"children":3039},{},[3040,3044,3049],{"type":45,"tag":147,"props":3041,"children":3042},{},[3043],{"type":51,"value":2762},{"type":45,"tag":147,"props":3045,"children":3046},{},[3047],{"type":51,"value":3048},"Actual Values",{"type":45,"tag":147,"props":3050,"children":3051},{},[3052],{"type":51,"value":2772},{"type":45,"tag":163,"props":3054,"children":3055},{},[3056,3074,3098,3122],{"type":45,"tag":143,"props":3057,"children":3058},{},[3059,3064,3069],{"type":45,"tag":170,"props":3060,"children":3061},{},[3062],{"type":51,"value":3063},"Vectorization width",{"type":45,"tag":170,"props":3065,"children":3066},{},[3067],{"type":51,"value":3068},"16 (fp32), 32 (bf16)",{"type":45,"tag":170,"props":3070,"children":3071},{},[3072],{"type":51,"value":3073},"Per-type VEC_ELEM_NUM",{"type":45,"tag":143,"props":3075,"children":3076},{},[3077,3082,3093],{"type":45,"tag":170,"props":3078,"children":3079},{},[3080],{"type":51,"value":3081},"Prefetch hint",{"type":45,"tag":170,"props":3083,"children":3084},{},[3085,3091],{"type":45,"tag":60,"props":3086,"children":3088},{"className":3087},[],[3089],{"type":51,"value":3090},"_MM_HINT_T1",{"type":51,"value":3092}," (L2)",{"type":45,"tag":170,"props":3094,"children":3095},{},[3096],{"type":51,"value":3097},"Different from GEMM (L1)",{"type":45,"tag":143,"props":3099,"children":3100},{},[3101,3106,3111],{"type":45,"tag":170,"props":3102,"children":3103},{},[3104],{"type":51,"value":3105},"OpenMP grain size",{"type":45,"tag":170,"props":3107,"children":3108},{},[3109],{"type":51,"value":3110},"1024",{"type":45,"tag":170,"props":3112,"children":3113},{},[3114,3120],{"type":45,"tag":60,"props":3115,"children":3117},{"className":3116},[],[3118],{"type":51,"value":3119},"at::parallel_for",{"type":51,"value":3121}," grain",{"type":45,"tag":143,"props":3123,"children":3124},{},[3125,3130,3141],{"type":45,"tag":170,"props":3126,"children":3127},{},[3128],{"type":51,"value":3129},"Threading",{"type":45,"tag":170,"props":3131,"children":3132},{},[3133,3139],{"type":45,"tag":60,"props":3134,"children":3136},{"className":3135},[],[3137],{"type":51,"value":3138},"#pragma omp parallel for",{"type":51,"value":3140}," over rows",{"type":45,"tag":170,"props":3142,"children":3143},{},[3144],{"type":51,"value":3145},"Simple 1D parallelism",{"type":45,"tag":510,"props":3147,"children":3149},{"id":3148},"phase-2-finalization",[3150],{"type":51,"value":3151},"Phase 2 Finalization",{"type":45,"tag":543,"props":3153,"children":3155},{"className":545,"code":3154,"language":547,"meta":548,"style":548},"python scripts\u002Ftrial_manager.py finalize \u003Ckernel_name> output\u002F\n# Re-run benchmark without cached baseline for final accurate comparison\npython scripts\u002Fbenchmark_cpu.py baseline.py --kernel-package \u003Cpkg> --op \u003Cfunc>\n",[3156],{"type":45,"tag":60,"props":3157,"children":3158},{"__ignoreMap":548},[3159,3194,3202],{"type":45,"tag":554,"props":3160,"children":3161},{"class":556,"line":557},[3162,3166,3170,3174,3178,3182,3186,3190],{"type":45,"tag":554,"props":3163,"children":3164},{"style":580},[3165],{"type":51,"value":583},{"type":45,"tag":554,"props":3167,"children":3168},{"style":586},[3169],{"type":51,"value":669},{"type":45,"tag":554,"props":3171,"children":3172},{"style":586},[3173],{"type":51,"value":914},{"type":45,"tag":554,"props":3175,"children":3176},{"style":597},[3177],{"type":51,"value":2036},{"type":45,"tag":554,"props":3179,"children":3180},{"style":586},[3181],{"type":51,"value":2041},{"type":45,"tag":554,"props":3183,"children":3184},{"style":758},[3185],{"type":51,"value":2046},{"type":45,"tag":554,"props":3187,"children":3188},{"style":597},[3189],{"type":51,"value":2051},{"type":45,"tag":554,"props":3191,"children":3192},{"style":586},[3193],{"type":51,"value":923},{"type":45,"tag":554,"props":3195,"children":3196},{"class":556,"line":567},[3197],{"type":45,"tag":554,"props":3198,"children":3199},{"style":561},[3200],{"type":51,"value":3201},"# Re-run benchmark without cached baseline for final accurate comparison\n",{"type":45,"tag":554,"props":3203,"children":3204},{"class":556,"line":576},[3205,3209,3213,3217,3221,3225,3229,3233,3237,3241,3245,3249,3253],{"type":45,"tag":554,"props":3206,"children":3207},{"style":580},[3208],{"type":51,"value":583},{"type":45,"tag":554,"props":3210,"children":3211},{"style":586},[3212],{"type":51,"value":810},{"type":45,"tag":554,"props":3214,"children":3215},{"style":586},[3216],{"type":51,"value":815},{"type":45,"tag":554,"props":3218,"children":3219},{"style":586},[3220],{"type":51,"value":820},{"type":45,"tag":554,"props":3222,"children":3223},{"style":597},[3224],{"type":51,"value":2036},{"type":45,"tag":554,"props":3226,"children":3227},{"style":586},[3228],{"type":51,"value":2165},{"type":45,"tag":554,"props":3230,"children":3231},{"style":758},[3232],{"type":51,"value":2170},{"type":45,"tag":554,"props":3234,"children":3235},{"style":597},[3236],{"type":51,"value":2051},{"type":45,"tag":554,"props":3238,"children":3239},{"style":586},[3240],{"type":51,"value":594},{"type":45,"tag":554,"props":3242,"children":3243},{"style":597},[3244],{"type":51,"value":2036},{"type":45,"tag":554,"props":3246,"children":3247},{"style":586},[3248],{"type":51,"value":2187},{"type":45,"tag":554,"props":3250,"children":3251},{"style":758},[3252],{"type":51,"value":20},{"type":45,"tag":554,"props":3254,"children":3255},{"style":597},[3256],{"type":51,"value":2196},{"type":45,"tag":116,"props":3258,"children":3260},{"id":3259},"reference-docs-read-during-phase-1",[3261],{"type":51,"value":3262},"Reference Docs — Read During Phase 1",{"type":45,"tag":135,"props":3264,"children":3265},{},[3266,3282],{"type":45,"tag":139,"props":3267,"children":3268},{},[3269],{"type":45,"tag":143,"props":3270,"children":3271},{},[3272,3277],{"type":45,"tag":147,"props":3273,"children":3274},{},[3275],{"type":51,"value":3276},"Doc",{"type":45,"tag":147,"props":3278,"children":3279},{},[3280],{"type":51,"value":3281},"Contents",{"type":45,"tag":163,"props":3283,"children":3284},{},[3285,3301,3318,3335],{"type":45,"tag":143,"props":3286,"children":3287},{},[3288,3296],{"type":45,"tag":170,"props":3289,"children":3290},{},[3291],{"type":45,"tag":60,"props":3292,"children":3294},{"className":3293},[],[3295],{"type":51,"value":1382},{"type":45,"tag":170,"props":3297,"children":3298},{},[3299],{"type":51,"value":3300},"cpu_features.hpp pattern, dispatch tiers",{"type":45,"tag":143,"props":3302,"children":3303},{},[3304,3313],{"type":45,"tag":170,"props":3305,"children":3306},{},[3307],{"type":45,"tag":60,"props":3308,"children":3310},{"className":3309},[],[3311],{"type":51,"value":3312},"references\u002Fbuild_system.yaml",{"type":45,"tag":170,"props":3314,"children":3315},{},[3316],{"type":51,"value":3317},"build.toml multi-target CPU compilation",{"type":45,"tag":143,"props":3319,"children":3320},{},[3321,3330],{"type":45,"tag":170,"props":3322,"children":3323},{},[3324],{"type":45,"tag":60,"props":3325,"children":3327},{"className":3326},[],[3328],{"type":51,"value":3329},"references\u002Fimplementation_reference.md",{"type":45,"tag":170,"props":3331,"children":3332},{},[3333],{"type":51,"value":3334},"C++ kernel templates, Unroll\u003CN>, tinygemm, torch_binding.cpp",{"type":45,"tag":143,"props":3336,"children":3337},{},[3338,3347],{"type":45,"tag":170,"props":3339,"children":3340},{},[3341],{"type":45,"tag":60,"props":3342,"children":3344},{"className":3343},[],[3345],{"type":51,"value":3346},"references\u002Fcorrectness.yaml",{"type":45,"tag":170,"props":3348,"children":3349},{},[3350],{"type":51,"value":3351},"Critical constraints: alignment, FTZ\u002FDAZ, denormals",{"type":45,"tag":116,"props":3353,"children":3355},{"id":3354},"reference-docs-read-during-phase-2",[3356],{"type":51,"value":3357},"Reference Docs — Read During Phase 2",{"type":45,"tag":135,"props":3359,"children":3360},{},[3361,3375],{"type":45,"tag":139,"props":3362,"children":3363},{},[3364],{"type":45,"tag":143,"props":3365,"children":3366},{},[3367,3371],{"type":45,"tag":147,"props":3368,"children":3369},{},[3370],{"type":51,"value":3276},{"type":45,"tag":147,"props":3372,"children":3373},{},[3374],{"type":51,"value":3281},{"type":45,"tag":163,"props":3376,"children":3377},{},[3378,3395,3412,3428,3445,3461,3478,3494,3511,3528],{"type":45,"tag":143,"props":3379,"children":3380},{},[3381,3390],{"type":45,"tag":170,"props":3382,"children":3383},{},[3384],{"type":45,"tag":60,"props":3385,"children":3387},{"className":3386},[],[3388],{"type":51,"value":3389},"references\u002Fsimd_optimization_patterns.yaml",{"type":45,"tag":170,"props":3391,"children":3392},{},[3393],{"type":51,"value":3394},"AVX2\u002FAVX512 vector abstractions and patterns",{"type":45,"tag":143,"props":3396,"children":3397},{},[3398,3407],{"type":45,"tag":170,"props":3399,"children":3400},{},[3401],{"type":45,"tag":60,"props":3402,"children":3404},{"className":3403},[],[3405],{"type":51,"value":3406},"references\u002Fquantized_gemm_patterns.yaml",{"type":45,"tag":170,"props":3408,"children":3409},{},[3410],{"type":51,"value":3411},"LUT + tinygemm + Unroll template for 4-bit GEMM",{"type":45,"tag":143,"props":3413,"children":3414},{},[3415,3423],{"type":45,"tag":170,"props":3416,"children":3417},{},[3418],{"type":45,"tag":60,"props":3419,"children":3421},{"className":3420},[],[3422],{"type":51,"value":1073},{"type":45,"tag":170,"props":3424,"children":3425},{},[3426],{"type":51,"value":3427},"brgemm API usage, VNNI packing, tinygemm vs brgemm selection (GEMM kernels only)",{"type":45,"tag":143,"props":3429,"children":3430},{},[3431,3440],{"type":45,"tag":170,"props":3432,"children":3433},{},[3434],{"type":45,"tag":60,"props":3435,"children":3437},{"className":3436},[],[3438],{"type":51,"value":3439},"references\u002Fmemory_patterns.yaml",{"type":45,"tag":170,"props":3441,"children":3442},{},[3443],{"type":51,"value":3444},"Prefetch, alignment, cache blocking",{"type":45,"tag":143,"props":3446,"children":3447},{},[3448,3456],{"type":45,"tag":170,"props":3449,"children":3450},{},[3451],{"type":45,"tag":60,"props":3452,"children":3454},{"className":3453},[],[3455],{"type":51,"value":1695},{"type":45,"tag":170,"props":3457,"children":3458},{},[3459],{"type":51,"value":3460},"OpenMP parallel patterns",{"type":45,"tag":143,"props":3462,"children":3463},{},[3464,3473],{"type":45,"tag":170,"props":3465,"children":3466},{},[3467],{"type":45,"tag":60,"props":3468,"children":3470},{"className":3469},[],[3471],{"type":51,"value":3472},"references\u002Fdtype_optimizations.yaml",{"type":45,"tag":170,"props":3474,"children":3475},{},[3476],{"type":51,"value":3477},"bf16\u002Ffp8\u002Fint8 handling and conversion on CPU",{"type":45,"tag":143,"props":3479,"children":3480},{},[3481,3489],{"type":45,"tag":170,"props":3482,"children":3483},{},[3484],{"type":45,"tag":60,"props":3485,"children":3487},{"className":3486},[],[3488],{"type":51,"value":2736},{"type":45,"tag":170,"props":3490,"children":3491},{},[3492],{"type":51,"value":3493},"Progressive L1→L5 optimization checklist + try-harder tree",{"type":45,"tag":143,"props":3495,"children":3496},{},[3497,3506],{"type":45,"tag":170,"props":3498,"children":3499},{},[3500],{"type":45,"tag":60,"props":3501,"children":3503},{"className":3502},[],[3504],{"type":51,"value":3505},"references\u002Foptimization_strategies.md",{"type":45,"tag":170,"props":3507,"children":3508},{},[3509],{"type":51,"value":3510},"Strategy reference, decision tree, checklist",{"type":45,"tag":143,"props":3512,"children":3513},{},[3514,3523],{"type":45,"tag":170,"props":3515,"children":3516},{},[3517],{"type":45,"tag":60,"props":3518,"children":3520},{"className":3519},[],[3521],{"type":51,"value":3522},"references\u002Fworkflow_details.md",{"type":45,"tag":170,"props":3524,"children":3525},{},[3526],{"type":51,"value":3527},"Detailed trial loop workflow",{"type":45,"tag":143,"props":3529,"children":3530},{},[3531,3540],{"type":45,"tag":170,"props":3532,"children":3533},{},[3534],{"type":45,"tag":60,"props":3535,"children":3537},{"className":3536},[],[3538],{"type":51,"value":3539},"references\u002Fhuggingface-kernels-integration.md",{"type":45,"tag":170,"props":3541,"children":3542},{},[3543],{"type":51,"value":3544},"Hub integration for CPU kernels",{"type":45,"tag":116,"props":3546,"children":3548},{"id":3547},"core-cpu-kernel-patterns",[3549],{"type":51,"value":3550},"Core CPU Kernel Patterns",{"type":45,"tag":510,"props":3552,"children":3554},{"id":3553},"runtime-dispatch-required-for-all-kernels",[3555],{"type":51,"value":3556},"Runtime Dispatch (Required for All Kernels)",{"type":45,"tag":54,"props":3558,"children":3559},{},[3560,3562,3567],{"type":51,"value":3561},"Every CPU kernel has its own ",{"type":45,"tag":60,"props":3563,"children":3565},{"className":3564},[],[3566],{"type":51,"value":1374},{"type":51,"value":3568}," (in its own namespace) and dispatches at runtime. Most kernels dispatch as AVX512 → fallback (no AVX2 tier):",{"type":45,"tag":543,"props":3570,"children":3574},{"className":3571,"code":3572,"language":3573,"meta":548,"style":548},"language-cpp shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\u002F\u002F my_kernel_cpu\u002Fcpu_features.hpp — each kernel has its OWN copy\nnamespace my_kernel_cpu {\nclass CPUFeatures {\npublic:\n    static bool hasAVX512BF16() { \u002F* CPUID + XCR0 checks *\u002F }\n    static bool hasAVX2() { \u002F* CPUID check *\u002F }\n    \u002F\u002F GEMM kernels also check: static bool hasAMX() { ... }\n};\n}\n\n\u002F\u002F my_kernel_cpu\u002Fmy_kernel_cpu.cpp — dispatcher\n#include \"cpu_features.hpp\"\n#include \"my_kernel_avx512.hpp\"\n\nvoid my_kernel(torch::Tensor& out, const torch::Tensor& input, ...) {\n    if (CPUFeatures::hasAVX512BF16()) {\n        avx512::my_kernel_impl(out, input, ...);\n    } else {\n        \u002F\u002F ATen fallback — inline or in a separate _fallback.cpp\n        out = torch::some_aten_op(input, ...);\n    }\n}\n","cpp",[3575],{"type":45,"tag":60,"props":3576,"children":3577},{"__ignoreMap":548},[3578,3586,3594,3602,3610,3618,3626,3634,3642,3650,3657,3665,3673,3681,3688,3696,3704,3712,3720,3728,3736,3745],{"type":45,"tag":554,"props":3579,"children":3580},{"class":556,"line":557},[3581],{"type":45,"tag":554,"props":3582,"children":3583},{},[3584],{"type":51,"value":3585},"\u002F\u002F my_kernel_cpu\u002Fcpu_features.hpp — each kernel has its OWN copy\n",{"type":45,"tag":554,"props":3587,"children":3588},{"class":556,"line":567},[3589],{"type":45,"tag":554,"props":3590,"children":3591},{},[3592],{"type":51,"value":3593},"namespace my_kernel_cpu {\n",{"type":45,"tag":554,"props":3595,"children":3596},{"class":556,"line":576},[3597],{"type":45,"tag":554,"props":3598,"children":3599},{},[3600],{"type":51,"value":3601},"class CPUFeatures {\n",{"type":45,"tag":554,"props":3603,"children":3604},{"class":556,"line":632},[3605],{"type":45,"tag":554,"props":3606,"children":3607},{},[3608],{"type":51,"value":3609},"public:\n",{"type":45,"tag":554,"props":3611,"children":3612},{"class":556,"line":641},[3613],{"type":45,"tag":554,"props":3614,"children":3615},{},[3616],{"type":51,"value":3617},"    static bool hasAVX512BF16() { \u002F* CPUID + XCR0 checks *\u002F }\n",{"type":45,"tag":554,"props":3619,"children":3620},{"class":556,"line":650},[3621],{"type":45,"tag":554,"props":3622,"children":3623},{},[3624],{"type":51,"value":3625},"    static bool hasAVX2() { \u002F* CPUID check *\u002F }\n",{"type":45,"tag":554,"props":3627,"children":3628},{"class":556,"line":659},[3629],{"type":45,"tag":554,"props":3630,"children":3631},{},[3632],{"type":51,"value":3633},"    \u002F\u002F GEMM kernels also check: static bool hasAMX() { ... }\n",{"type":45,"tag":554,"props":3635,"children":3636},{"class":556,"line":687},[3637],{"type":45,"tag":554,"props":3638,"children":3639},{},[3640],{"type":51,"value":3641},"};\n",{"type":45,"tag":554,"props":3643,"children":3644},{"class":556,"line":695},[3645],{"type":45,"tag":554,"props":3646,"children":3647},{},[3648],{"type":51,"value":3649},"}\n",{"type":45,"tag":554,"props":3651,"children":3652},{"class":556,"line":704},[3653],{"type":45,"tag":554,"props":3654,"children":3655},{"emptyLinePlaceholder":38},[3656],{"type":51,"value":638},{"type":45,"tag":554,"props":3658,"children":3659},{"class":556,"line":774},[3660],{"type":45,"tag":554,"props":3661,"children":3662},{},[3663],{"type":51,"value":3664},"\u002F\u002F my_kernel_cpu\u002Fmy_kernel_cpu.cpp — dispatcher\n",{"type":45,"tag":554,"props":3666,"children":3667},{"class":556,"line":782},[3668],{"type":45,"tag":554,"props":3669,"children":3670},{},[3671],{"type":51,"value":3672},"#include \"cpu_features.hpp\"\n",{"type":45,"tag":554,"props":3674,"children":3675},{"class":556,"line":791},[3676],{"type":45,"tag":554,"props":3677,"children":3678},{},[3679],{"type":51,"value":3680},"#include \"my_kernel_avx512.hpp\"\n",{"type":45,"tag":554,"props":3682,"children":3683},{"class":556,"line":800},[3684],{"type":45,"tag":554,"props":3685,"children":3686},{"emptyLinePlaceholder":38},[3687],{"type":51,"value":638},{"type":45,"tag":554,"props":3689,"children":3690},{"class":556,"line":837},[3691],{"type":45,"tag":554,"props":3692,"children":3693},{},[3694],{"type":51,"value":3695},"void my_kernel(torch::Tensor& out, const torch::Tensor& input, ...) {\n",{"type":45,"tag":554,"props":3697,"children":3698},{"class":556,"line":845},[3699],{"type":45,"tag":554,"props":3700,"children":3701},{},[3702],{"type":51,"value":3703},"    if (CPUFeatures::hasAVX512BF16()) {\n",{"type":45,"tag":554,"props":3705,"children":3706},{"class":556,"line":854},[3707],{"type":45,"tag":554,"props":3708,"children":3709},{},[3710],{"type":51,"value":3711},"        avx512::my_kernel_impl(out, input, ...);\n",{"type":45,"tag":554,"props":3713,"children":3714},{"class":556,"line":883},[3715],{"type":45,"tag":554,"props":3716,"children":3717},{},[3718],{"type":51,"value":3719},"    } else {\n",{"type":45,"tag":554,"props":3721,"children":3722},{"class":556,"line":891},[3723],{"type":45,"tag":554,"props":3724,"children":3725},{},[3726],{"type":51,"value":3727},"        \u002F\u002F ATen fallback — inline or in a separate _fallback.cpp\n",{"type":45,"tag":554,"props":3729,"children":3730},{"class":556,"line":900},[3731],{"type":45,"tag":554,"props":3732,"children":3733},{},[3734],{"type":51,"value":3735},"        out = torch::some_aten_op(input, ...);\n",{"type":45,"tag":554,"props":3737,"children":3739},{"class":556,"line":3738},21,[3740],{"type":45,"tag":554,"props":3741,"children":3742},{},[3743],{"type":51,"value":3744},"    }\n",{"type":45,"tag":554,"props":3746,"children":3748},{"class":556,"line":3747},22,[3749],{"type":45,"tag":554,"props":3750,"children":3751},{},[3752],{"type":51,"value":3649},{"type":45,"tag":69,"props":3754,"children":3755},{},[3756],{"type":45,"tag":54,"props":3757,"children":3758},{},[3759,3763,3765,3771],{"type":45,"tag":76,"props":3760,"children":3761},{},[3762],{"type":51,"value":1089},{"type":51,"value":3764},": Only rmsnorm has a three-tier dispatch (AVX512 → AVX2 → ATen). GEMM kernels skip AVX2. Flash-attn2 additionally requires AMX via ",{"type":45,"tag":60,"props":3766,"children":3768},{"className":3767},[],[3769],{"type":51,"value":3770},"hasAllRequiredFeatures()",{"type":51,"value":114},{"type":45,"tag":69,"props":3773,"children":3774},{},[3775],{"type":45,"tag":54,"props":3776,"children":3777},{},[3778,3780],{"type":51,"value":3779},"Full pattern: ",{"type":45,"tag":1071,"props":3781,"children":3782},{"href":1382},[3783],{"type":51,"value":3784},"runtime_dispatch.yaml",{"type":45,"tag":510,"props":3786,"children":3788},{"id":3787},"buildtoml-multi-target-compilation",[3789],{"type":51,"value":3790},"build.toml Multi-Target Compilation",{"type":45,"tag":54,"props":3792,"children":3793},{},[3794,3796,3802,3804,3810],{"type":51,"value":3795},"Each SIMD tier is a separate ",{"type":45,"tag":60,"props":3797,"children":3799},{"className":3798},[],[3800],{"type":51,"value":3801},"[kernel.*]",{"type":51,"value":3803}," section with its own compiler flags. The ",{"type":45,"tag":60,"props":3805,"children":3807},{"className":3806},[],[3808],{"type":51,"value":3809},"include",{"type":51,"value":3811}," directive is required for header resolution:",{"type":45,"tag":543,"props":3813,"children":3817},{"className":3814,"code":3815,"language":3816,"meta":548,"style":548},"language-toml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[kernel.my_kernel_cpu]\nbackend = \"cpu\"\ndepends = [\"torch\"]\ninclude = [\"my_kernel_cpu\"]\nsrc = [\n    \"my_kernel_cpu\u002Fmy_kernel_cpu.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_cpu_torch.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_cpu.hpp\",\n    \"my_kernel_cpu\u002Fcpu_features.hpp\",\n]\n\n[kernel.my_kernel_cpu_avx512]\nbackend = \"cpu\"\n# Note: For GEMM kernels (e.g., flash-attn2, megablocks), you must also include \"-mamx-tile\", \"-mamx-bf16\", \"-mamx-int8\"\ncxx-flags = [\"-mavx512f\", \"-mavx512bf16\", \"-mavx512vl\", \"-mavx512dq\", \"-mavx512bw\", \"-mavx512vbmi\", \"-mfma\", \"-mf16c\", \"-fopenmp\"]\ndepends = [\"torch\"]\ninclude = [\"my_kernel_cpu\"]\nsrc = [\n    \"my_kernel_cpu\u002Fmy_kernel_avx512.cpp\",\n    \"my_kernel_cpu\u002Fmy_kernel_avx512.hpp\",\n]\n","toml",[3818],{"type":45,"tag":60,"props":3819,"children":3820},{"__ignoreMap":548},[3821,3829,3837,3845,3853,3861,3869,3877,3885,3893,3901,3908,3916,3923,3931,3939,3946,3953,3960,3968,3976],{"type":45,"tag":554,"props":3822,"children":3823},{"class":556,"line":557},[3824],{"type":45,"tag":554,"props":3825,"children":3826},{},[3827],{"type":51,"value":3828},"[kernel.my_kernel_cpu]\n",{"type":45,"tag":554,"props":3830,"children":3831},{"class":556,"line":567},[3832],{"type":45,"tag":554,"props":3833,"children":3834},{},[3835],{"type":51,"value":3836},"backend = \"cpu\"\n",{"type":45,"tag":554,"props":3838,"children":3839},{"class":556,"line":576},[3840],{"type":45,"tag":554,"props":3841,"children":3842},{},[3843],{"type":51,"value":3844},"depends = [\"torch\"]\n",{"type":45,"tag":554,"props":3846,"children":3847},{"class":556,"line":632},[3848],{"type":45,"tag":554,"props":3849,"children":3850},{},[3851],{"type":51,"value":3852},"include = [\"my_kernel_cpu\"]\n",{"type":45,"tag":554,"props":3854,"children":3855},{"class":556,"line":641},[3856],{"type":45,"tag":554,"props":3857,"children":3858},{},[3859],{"type":51,"value":3860},"src = [\n",{"type":45,"tag":554,"props":3862,"children":3863},{"class":556,"line":650},[3864],{"type":45,"tag":554,"props":3865,"children":3866},{},[3867],{"type":51,"value":3868},"    \"my_kernel_cpu\u002Fmy_kernel_cpu.cpp\",\n",{"type":45,"tag":554,"props":3870,"children":3871},{"class":556,"line":659},[3872],{"type":45,"tag":554,"props":3873,"children":3874},{},[3875],{"type":51,"value":3876},"    \"my_kernel_cpu\u002Fmy_kernel_cpu_torch.cpp\",\n",{"type":45,"tag":554,"props":3878,"children":3879},{"class":556,"line":687},[3880],{"type":45,"tag":554,"props":3881,"children":3882},{},[3883],{"type":51,"value":3884},"    \"my_kernel_cpu\u002Fmy_kernel_cpu.hpp\",\n",{"type":45,"tag":554,"props":3886,"children":3887},{"class":556,"line":695},[3888],{"type":45,"tag":554,"props":3889,"children":3890},{},[3891],{"type":51,"value":3892},"    \"my_kernel_cpu\u002Fcpu_features.hpp\",\n",{"type":45,"tag":554,"props":3894,"children":3895},{"class":556,"line":704},[3896],{"type":45,"tag":554,"props":3897,"children":3898},{},[3899],{"type":51,"value":3900},"]\n",{"type":45,"tag":554,"props":3902,"children":3903},{"class":556,"line":774},[3904],{"type":45,"tag":554,"props":3905,"children":3906},{"emptyLinePlaceholder":38},[3907],{"type":51,"value":638},{"type":45,"tag":554,"props":3909,"children":3910},{"class":556,"line":782},[3911],{"type":45,"tag":554,"props":3912,"children":3913},{},[3914],{"type":51,"value":3915},"[kernel.my_kernel_cpu_avx512]\n",{"type":45,"tag":554,"props":3917,"children":3918},{"class":556,"line":791},[3919],{"type":45,"tag":554,"props":3920,"children":3921},{},[3922],{"type":51,"value":3836},{"type":45,"tag":554,"props":3924,"children":3925},{"class":556,"line":800},[3926],{"type":45,"tag":554,"props":3927,"children":3928},{},[3929],{"type":51,"value":3930},"# Note: For GEMM kernels (e.g., flash-attn2, megablocks), you must also include \"-mamx-tile\", \"-mamx-bf16\", \"-mamx-int8\"\n",{"type":45,"tag":554,"props":3932,"children":3933},{"class":556,"line":837},[3934],{"type":45,"tag":554,"props":3935,"children":3936},{},[3937],{"type":51,"value":3938},"cxx-flags = [\"-mavx512f\", \"-mavx512bf16\", \"-mavx512vl\", \"-mavx512dq\", \"-mavx512bw\", \"-mavx512vbmi\", \"-mfma\", \"-mf16c\", \"-fopenmp\"]\n",{"type":45,"tag":554,"props":3940,"children":3941},{"class":556,"line":845},[3942],{"type":45,"tag":554,"props":3943,"children":3944},{},[3945],{"type":51,"value":3844},{"type":45,"tag":554,"props":3947,"children":3948},{"class":556,"line":854},[3949],{"type":45,"tag":554,"props":3950,"children":3951},{},[3952],{"type":51,"value":3852},{"type":45,"tag":554,"props":3954,"children":3955},{"class":556,"line":883},[3956],{"type":45,"tag":554,"props":3957,"children":3958},{},[3959],{"type":51,"value":3860},{"type":45,"tag":554,"props":3961,"children":3962},{"class":556,"line":891},[3963],{"type":45,"tag":554,"props":3964,"children":3965},{},[3966],{"type":51,"value":3967},"    \"my_kernel_cpu\u002Fmy_kernel_avx512.cpp\",\n",{"type":45,"tag":554,"props":3969,"children":3970},{"class":556,"line":900},[3971],{"type":45,"tag":554,"props":3972,"children":3973},{},[3974],{"type":51,"value":3975},"    \"my_kernel_cpu\u002Fmy_kernel_avx512.hpp\",\n",{"type":45,"tag":554,"props":3977,"children":3978},{"class":556,"line":3738},[3979],{"type":45,"tag":554,"props":3980,"children":3981},{},[3982],{"type":51,"value":3900},{"type":45,"tag":69,"props":3984,"children":3985},{},[3986],{"type":45,"tag":54,"props":3987,"children":3988},{},[3989,3993,3995,4001,4003,4009],{"type":45,"tag":76,"props":3990,"children":3991},{},[3992],{"type":51,"value":1089},{"type":51,"value":3994},": Every section needs ",{"type":45,"tag":60,"props":3996,"children":3998},{"className":3997},[],[3999],{"type":51,"value":4000},"include = [\"\u003Ckernel_dir>\"]",{"type":51,"value":4002}," for header resolution. The ",{"type":45,"tag":60,"props":4004,"children":4006},{"className":4005},[],[4007],{"type":51,"value":4008},"_torch.cpp",{"type":51,"value":4010}," file bridges Python-facing declarations to the C++ dispatcher. AVX2 section is optional (only rmsnorm has one).",{"type":45,"tag":69,"props":4012,"children":4013},{},[4014],{"type":45,"tag":54,"props":4015,"children":4016},{},[4017,4018],{"type":51,"value":3779},{"type":45,"tag":1071,"props":4019,"children":4020},{"href":3312},[4021],{"type":51,"value":4022},"build_system.yaml",{"type":45,"tag":510,"props":4024,"children":4026},{"id":4025},"torch_bindingcpp-registration",[4027],{"type":51,"value":4028},"torch_binding.cpp Registration",{"type":45,"tag":54,"props":4030,"children":4031},{},[4032,4034,4040],{"type":51,"value":4033},"All kernels use ",{"type":45,"tag":60,"props":4035,"children":4037},{"className":4036},[],[4038],{"type":51,"value":4039},"registration.h",{"type":51,"value":4041}," macros for op registration:",{"type":45,"tag":543,"props":4043,"children":4045},{"className":3571,"code":4044,"language":3573,"meta":548,"style":548},"#include \"registration.h\"\n\n\u002F\u002F Forward declarations\n#if defined(CPU_KERNEL)\ntorch::Tensor my_kernel_cpu_forward(torch::Tensor input, torch::Tensor weight, float eps);\n#endif\n\nTORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n    ops.def(\"forward(Tensor input, Tensor weight, float eps) -> Tensor\");\n    ops.impl(\"forward\", torch::kCPU, &my_kernel_cpu_forward);\n}\n\nREGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n",[4046],{"type":45,"tag":60,"props":4047,"children":4048},{"__ignoreMap":548},[4049,4057,4064,4072,4080,4088,4096,4103,4111,4119,4127,4134,4141],{"type":45,"tag":554,"props":4050,"children":4051},{"class":556,"line":557},[4052],{"type":45,"tag":554,"props":4053,"children":4054},{},[4055],{"type":51,"value":4056},"#include \"registration.h\"\n",{"type":45,"tag":554,"props":4058,"children":4059},{"class":556,"line":567},[4060],{"type":45,"tag":554,"props":4061,"children":4062},{"emptyLinePlaceholder":38},[4063],{"type":51,"value":638},{"type":45,"tag":554,"props":4065,"children":4066},{"class":556,"line":576},[4067],{"type":45,"tag":554,"props":4068,"children":4069},{},[4070],{"type":51,"value":4071},"\u002F\u002F Forward declarations\n",{"type":45,"tag":554,"props":4073,"children":4074},{"class":556,"line":632},[4075],{"type":45,"tag":554,"props":4076,"children":4077},{},[4078],{"type":51,"value":4079},"#if defined(CPU_KERNEL)\n",{"type":45,"tag":554,"props":4081,"children":4082},{"class":556,"line":641},[4083],{"type":45,"tag":554,"props":4084,"children":4085},{},[4086],{"type":51,"value":4087},"torch::Tensor my_kernel_cpu_forward(torch::Tensor input, torch::Tensor weight, float eps);\n",{"type":45,"tag":554,"props":4089,"children":4090},{"class":556,"line":650},[4091],{"type":45,"tag":554,"props":4092,"children":4093},{},[4094],{"type":51,"value":4095},"#endif\n",{"type":45,"tag":554,"props":4097,"children":4098},{"class":556,"line":659},[4099],{"type":45,"tag":554,"props":4100,"children":4101},{"emptyLinePlaceholder":38},[4102],{"type":51,"value":638},{"type":45,"tag":554,"props":4104,"children":4105},{"class":556,"line":687},[4106],{"type":45,"tag":554,"props":4107,"children":4108},{},[4109],{"type":51,"value":4110},"TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {\n",{"type":45,"tag":554,"props":4112,"children":4113},{"class":556,"line":695},[4114],{"type":45,"tag":554,"props":4115,"children":4116},{},[4117],{"type":51,"value":4118},"    ops.def(\"forward(Tensor input, Tensor weight, float eps) -> Tensor\");\n",{"type":45,"tag":554,"props":4120,"children":4121},{"class":556,"line":704},[4122],{"type":45,"tag":554,"props":4123,"children":4124},{},[4125],{"type":51,"value":4126},"    ops.impl(\"forward\", torch::kCPU, &my_kernel_cpu_forward);\n",{"type":45,"tag":554,"props":4128,"children":4129},{"class":556,"line":774},[4130],{"type":45,"tag":554,"props":4131,"children":4132},{},[4133],{"type":51,"value":3649},{"type":45,"tag":554,"props":4135,"children":4136},{"class":556,"line":782},[4137],{"type":45,"tag":554,"props":4138,"children":4139},{"emptyLinePlaceholder":38},[4140],{"type":51,"value":638},{"type":45,"tag":554,"props":4142,"children":4143},{"class":556,"line":791},[4144],{"type":45,"tag":554,"props":4145,"children":4146},{},[4147],{"type":51,"value":4148},"REGISTER_EXTENSION(TORCH_EXTENSION_NAME)\n",{"type":45,"tag":69,"props":4150,"children":4151},{},[4152],{"type":45,"tag":54,"props":4153,"children":4154},{},[4155,4159,4161,4166,4168,4174,4176,4182],{"type":45,"tag":76,"props":4156,"children":4157},{},[4158],{"type":51,"value":1089},{"type":51,"value":4160},": ",{"type":45,"tag":60,"props":4162,"children":4164},{"className":4163},[],[4165],{"type":51,"value":4039},{"type":51,"value":4167}," is provided by kernel-builder. Multi-device kernels (rmsnorm, megablocks) use ",{"type":45,"tag":60,"props":4169,"children":4171},{"className":4170},[],[4172],{"type":51,"value":4173},"#if defined(CPU_KERNEL)",{"type":51,"value":4175}," \u002F ",{"type":45,"tag":60,"props":4177,"children":4179},{"className":4178},[],[4180],{"type":51,"value":4181},"#elif defined(CUDA_KERNEL)",{"type":51,"value":4183}," guards.",{"type":45,"tag":510,"props":4185,"children":4187},{"id":4186},"vector-type-abstractions-avx512",[4188],{"type":51,"value":4189},"Vector Type Abstractions (AVX512)",{"type":45,"tag":54,"props":4191,"children":4192},{},[4193],{"type":51,"value":4194},"Wrap raw intrinsics in typed vector classes for readability:",{"type":45,"tag":543,"props":4196,"children":4198},{"className":3571,"code":4197,"language":3573,"meta":548,"style":548},"\u002F\u002F cpu_types_avx512.hpp\nstruct FP32Vec16 {\n    __m512 reg;\n    FP32Vec16(float v) : reg(_mm512_set1_ps(v)) {}\n    FP32Vec16(__m512 r) : reg(r) {}\n    FP32Vec16 operator*(const FP32Vec16& other) const {\n        return FP32Vec16(_mm512_mul_ps(reg, other.reg));\n    }\n    float reduce_sum() const { return _mm512_reduce_add_ps(reg); }\n};\n",[4199],{"type":45,"tag":60,"props":4200,"children":4201},{"__ignoreMap":548},[4202,4210,4218,4226,4234,4242,4250,4258,4265,4273],{"type":45,"tag":554,"props":4203,"children":4204},{"class":556,"line":557},[4205],{"type":45,"tag":554,"props":4206,"children":4207},{},[4208],{"type":51,"value":4209},"\u002F\u002F cpu_types_avx512.hpp\n",{"type":45,"tag":554,"props":4211,"children":4212},{"class":556,"line":567},[4213],{"type":45,"tag":554,"props":4214,"children":4215},{},[4216],{"type":51,"value":4217},"struct FP32Vec16 {\n",{"type":45,"tag":554,"props":4219,"children":4220},{"class":556,"line":576},[4221],{"type":45,"tag":554,"props":4222,"children":4223},{},[4224],{"type":51,"value":4225},"    __m512 reg;\n",{"type":45,"tag":554,"props":4227,"children":4228},{"class":556,"line":632},[4229],{"type":45,"tag":554,"props":4230,"children":4231},{},[4232],{"type":51,"value":4233},"    FP32Vec16(float v) : reg(_mm512_set1_ps(v)) {}\n",{"type":45,"tag":554,"props":4235,"children":4236},{"class":556,"line":641},[4237],{"type":45,"tag":554,"props":4238,"children":4239},{},[4240],{"type":51,"value":4241},"    FP32Vec16(__m512 r) : reg(r) {}\n",{"type":45,"tag":554,"props":4243,"children":4244},{"class":556,"line":650},[4245],{"type":45,"tag":554,"props":4246,"children":4247},{},[4248],{"type":51,"value":4249},"    FP32Vec16 operator*(const FP32Vec16& other) const {\n",{"type":45,"tag":554,"props":4251,"children":4252},{"class":556,"line":659},[4253],{"type":45,"tag":554,"props":4254,"children":4255},{},[4256],{"type":51,"value":4257},"        return FP32Vec16(_mm512_mul_ps(reg, other.reg));\n",{"type":45,"tag":554,"props":4259,"children":4260},{"class":556,"line":687},[4261],{"type":45,"tag":554,"props":4262,"children":4263},{},[4264],{"type":51,"value":3744},{"type":45,"tag":554,"props":4266,"children":4267},{"class":556,"line":695},[4268],{"type":45,"tag":554,"props":4269,"children":4270},{},[4271],{"type":51,"value":4272},"    float reduce_sum() const { return _mm512_reduce_add_ps(reg); }\n",{"type":45,"tag":554,"props":4274,"children":4275},{"class":556,"line":704},[4276],{"type":45,"tag":554,"props":4277,"children":4278},{},[4279],{"type":51,"value":3641},{"type":45,"tag":69,"props":4281,"children":4282},{},[4283],{"type":45,"tag":54,"props":4284,"children":4285},{},[4286,4287],{"type":51,"value":3779},{"type":45,"tag":1071,"props":4288,"children":4289},{"href":3389},[4290],{"type":51,"value":4291},"simd_optimization_patterns.yaml",{"type":45,"tag":510,"props":4293,"children":4295},{"id":4294},"quantized-gemm-template-int4nf4fp4",[4296],{"type":51,"value":4297},"Quantized GEMM Template (INT4\u002FNF4\u002FFP4)",{"type":45,"tag":54,"props":4299,"children":4300},{},[4301],{"type":51,"value":4302},"All 4-bit quantized GEMM kernels share the same skeleton — only the LUT and zero-point handling differ:",{"type":45,"tag":543,"props":4304,"children":4308},{"className":4305,"code":4307,"language":51},[4306],"language-text","nibble split → zero subtract → LUT lookup → _mm512_dpbf16_ps accumulate → scale fmadd (per group) → bf16 output\n",[4309],{"type":45,"tag":60,"props":4310,"children":4311},{"__ignoreMap":548},[4312],{"type":51,"value":4307},{"type":45,"tag":54,"props":4314,"children":4315},{},[4316],{"type":51,"value":4317},"The parameterized components:",{"type":45,"tag":1104,"props":4319,"children":4320},{},[4321,4331,4341,4351,4478,4508],{"type":45,"tag":1108,"props":4322,"children":4323},{},[4324,4329],{"type":45,"tag":76,"props":4325,"children":4326},{},[4327],{"type":51,"value":4328},"LUT",{"type":51,"value":4330},": GPTQ (linear INT4), BnB (NF4\u002FFP4), MegaBlocks (FP8\u002FMXFP4)",{"type":45,"tag":1108,"props":4332,"children":4333},{},[4334,4339],{"type":45,"tag":76,"props":4335,"children":4336},{},[4337],{"type":51,"value":4338},"Zero-point",{"type":51,"value":4340},": per-group (GPTQ), none\u002Fencoded in LUT (BnB), per-block (FP8)",{"type":45,"tag":1108,"props":4342,"children":4343},{},[4344,4349],{"type":45,"tag":76,"props":4345,"children":4346},{},[4347],{"type":51,"value":4348},"Algorithm",{"type":51,"value":4350},": tinygemm (small M, fused) vs brgemm (large M, unpack+BLAS)",{"type":45,"tag":1108,"props":4352,"children":4353},{},[4354,4359,4361],{"type":45,"tag":76,"props":4355,"children":4356},{},[4357],{"type":51,"value":4358},"Weight conversion",{"type":51,"value":4360},": The C++ kernel expects a specific block-interleaved format, NOT raw checkpoint format. Each framework converts in its own repo:\n",{"type":45,"tag":1104,"props":4362,"children":4363},{},[4364,4410,4446],{"type":45,"tag":1108,"props":4365,"children":4366},{},[4367,4372,4373,4379,4381,4386,4388,4394,4396,4401,4403,4408],{"type":45,"tag":76,"props":4368,"children":4369},{},[4370],{"type":51,"value":4371},"GPTQ",{"type":51,"value":4160},{"type":45,"tag":60,"props":4374,"children":4376},{"className":4375},[],[4377],{"type":51,"value":4378},"transform_cpu()",{"type":51,"value":4380}," unpacks int32→uint8, reorders by g_idx, transposes to ",{"type":45,"tag":554,"props":4382,"children":4383},{},[4384],{"type":51,"value":4385},"N,K",{"type":51,"value":4387},"; then ",{"type":45,"tag":60,"props":4389,"children":4391},{"className":4390},[],[4392],{"type":51,"value":4393},"convert_weight_packed_zp()",{"type":51,"value":4395}," repacks to ",{"type":45,"tag":554,"props":4397,"children":4398},{},[4399],{"type":51,"value":4400},"N,K\u002F2",{"type":51,"value":4402}," block-interleaved (BLOCK_N=32). Zeros unpacked to ",{"type":45,"tag":554,"props":4404,"children":4405},{},[4406],{"type":51,"value":4407},"groups,N",{"type":51,"value":4409}," uint8. Scales to bf16. Done at first forward in GPTQModel repo.",{"type":45,"tag":1108,"props":4411,"children":4412},{},[4413,4418,4419,4425,4427,4431,4433,4437,4439,4444],{"type":45,"tag":76,"props":4414,"children":4415},{},[4416],{"type":51,"value":4417},"BnB",{"type":51,"value":4160},{"type":45,"tag":60,"props":4420,"children":4422},{"className":4421},[],[4423],{"type":51,"value":4424},"_convert_weight_packed_for_cpu()",{"type":51,"value":4426}," unpacks uint8 nibbles→",{"type":45,"tag":554,"props":4428,"children":4429},{},[4430],{"type":51,"value":4385},{"type":51,"value":4432},", repacks to ",{"type":45,"tag":554,"props":4434,"children":4435},{},[4436],{"type":51,"value":4400},{"type":51,"value":4438}," block-interleaved (same algo as GPTQ). Denests nested absmax. Transposes scales to ",{"type":45,"tag":554,"props":4440,"children":4441},{},[4442],{"type":51,"value":4443},"K\u002Fblocksize,N",{"type":51,"value":4445}," bf16. Done at first forward in bitsandbytes repo.",{"type":45,"tag":1108,"props":4447,"children":4448},{},[4449,4454,4455,4461,4463,4469,4471,4477],{"type":45,"tag":76,"props":4450,"children":4451},{},[4452],{"type":51,"value":4453},"Megablocks MoE",{"type":51,"value":4160},{"type":45,"tag":60,"props":4456,"children":4458},{"className":4457},[],[4459],{"type":51,"value":4460},"ops.convert_weight_packed()",{"type":51,"value":4462}," does transpose+VNNI pack. ",{"type":45,"tag":60,"props":4464,"children":4466},{"className":4465},[],[4467],{"type":51,"value":4468},"ops.convert_scale_packed()",{"type":51,"value":4470}," reorders scales. Cached via ",{"type":45,"tag":60,"props":4472,"children":4474},{"className":4473},[],[4475],{"type":51,"value":4476},"packed_weight=True",{"type":51,"value":114},{"type":45,"tag":1108,"props":4479,"children":4480},{},[4481,4486,4488],{"type":45,"tag":76,"props":4482,"children":4483},{},[4484],{"type":51,"value":4485},"VNNI Conversion (K\u002FV Activations)",{"type":51,"value":4487},":\n",{"type":45,"tag":1104,"props":4489,"children":4490},{},[4491],{"type":45,"tag":1108,"props":4492,"children":4493},{},[4494,4499,4500,4506],{"type":45,"tag":76,"props":4495,"children":4496},{},[4497],{"type":51,"value":4498},"Flash Attention",{"type":51,"value":4160},{"type":45,"tag":60,"props":4501,"children":4503},{"className":4502},[],[4504],{"type":51,"value":4505},"pack_vnni()",{"type":51,"value":4507}," per tile per forward (K\u002FV change every call, so caching is not possible).",{"type":45,"tag":1108,"props":4509,"children":4510},{},[4511,4516],{"type":45,"tag":76,"props":4512,"children":4513},{},[4514],{"type":51,"value":4515},"Element-wise (RMSNorm)",{"type":51,"value":4517},": No conversion needed.",{"type":45,"tag":69,"props":4519,"children":4520},{},[4521],{"type":45,"tag":54,"props":4522,"children":4523},{},[4524,4525,4530,4532],{"type":51,"value":3779},{"type":45,"tag":1071,"props":4526,"children":4527},{"href":3406},[4528],{"type":51,"value":4529},"quantized_gemm_patterns.yaml",{"type":51,"value":4531},", weight conversion: ",{"type":45,"tag":1071,"props":4533,"children":4534},{"href":1073},[4535],{"type":51,"value":1076},{"type":45,"tag":116,"props":4537,"children":4539},{"id":4538},"critical-cpu-constraints",[4540],{"type":51,"value":4541},"Critical CPU Constraints",{"type":45,"tag":1104,"props":4543,"children":4544},{},[4545,4570,4588,4598,4624,4640],{"type":45,"tag":1108,"props":4546,"children":4547},{},[4548,4553,4555,4561,4563,4569],{"type":45,"tag":76,"props":4549,"children":4550},{},[4551],{"type":51,"value":4552},"Always use unaligned loads",{"type":51,"value":4554},": All existing kernels use ",{"type":45,"tag":60,"props":4556,"children":4558},{"className":4557},[],[4559],{"type":51,"value":4560},"_mm512_loadu_*",{"type":51,"value":4562}," exclusively. Never use ",{"type":45,"tag":60,"props":4564,"children":4566},{"className":4565},[],[4567],{"type":51,"value":4568},"_mm512_load_*",{"type":51,"value":114},{"type":45,"tag":1108,"props":4571,"children":4572},{},[4573,4578,4580,4586],{"type":45,"tag":76,"props":4574,"children":4575},{},[4576],{"type":51,"value":4577},"Edge cases",{"type":51,"value":4579},": When ",{"type":45,"tag":60,"props":4581,"children":4583},{"className":4582},[],[4584],{"type":51,"value":4585},"hidden_size % VEC_ELEM_NUM != 0",{"type":51,"value":4587},", handle the tail with scalar or masked SIMD ops.",{"type":45,"tag":1108,"props":4589,"children":4590},{},[4591,4596],{"type":45,"tag":76,"props":4592,"children":4593},{},[4594],{"type":51,"value":4595},"FTZ\u002FDAZ",{"type":51,"value":4597},": Flush-to-zero and denormals-as-zero may be set by PyTorch. Do NOT assume IEEE 754 denormal behavior.",{"type":45,"tag":1108,"props":4599,"children":4600},{},[4601,4606,4608,4614,4616,4622],{"type":45,"tag":76,"props":4602,"children":4603},{},[4604],{"type":51,"value":4605},"OpenMP overhead",{"type":51,"value":4607},": For small tensors, use ",{"type":45,"tag":60,"props":4609,"children":4611},{"className":4610},[],[4612],{"type":51,"value":4613},"adjust_num_threads(m)",{"type":51,"value":4615}," to reduce thread count. GEMM kernels use ",{"type":45,"tag":60,"props":4617,"children":4619},{"className":4618},[],[4620],{"type":51,"value":4621},"parallel_2d",{"type":51,"value":4623}," for 2D thread decomposition.",{"type":45,"tag":1108,"props":4625,"children":4626},{},[4627,4632,4633,4638],{"type":45,"tag":76,"props":4628,"children":4629},{},[4630],{"type":51,"value":4631},"bf16 precision",{"type":51,"value":4160},{"type":45,"tag":60,"props":4634,"children":4636},{"className":4635},[],[4637],{"type":51,"value":1023},{"type":51,"value":4639}," accumulates in fp32 but inputs are bf16 — precision loss is expected. Use atol=1e-2 for correctness checks.",{"type":45,"tag":1108,"props":4641,"children":4642},{},[4643,4648,4650,4656],{"type":45,"tag":76,"props":4644,"children":4645},{},[4646],{"type":51,"value":4647},"Data alignment",{"type":51,"value":4649},": Use ",{"type":45,"tag":60,"props":4651,"children":4653},{"className":4652},[],[4654],{"type":51,"value":4655},"alignas(64)",{"type":51,"value":4657}," for stack-allocated tile buffers to optimize cache-line access.",{"type":45,"tag":69,"props":4659,"children":4660},{},[4661],{"type":45,"tag":54,"props":4662,"children":4663},{},[4664,4666],{"type":51,"value":4665},"Full constraint list: ",{"type":45,"tag":1071,"props":4667,"children":4668},{"href":3346},[4669],{"type":51,"value":4670},"correctness.yaml",{"type":45,"tag":116,"props":4672,"children":4674},{"id":4673},"common-issues",[4675],{"type":51,"value":4676},"Common Issues",{"type":45,"tag":135,"props":4678,"children":4679},{},[4680,4701],{"type":45,"tag":139,"props":4681,"children":4682},{},[4683],{"type":45,"tag":143,"props":4684,"children":4685},{},[4686,4691,4696],{"type":45,"tag":147,"props":4687,"children":4688},{},[4689],{"type":51,"value":4690},"Issue",{"type":45,"tag":147,"props":4692,"children":4693},{},[4694],{"type":51,"value":4695},"Symptom",{"type":45,"tag":147,"props":4697,"children":4698},{},[4699],{"type":51,"value":4700},"Fix",{"type":45,"tag":163,"props":4702,"children":4703},{},[4704,4737,4758,4787,4816,4888],{"type":45,"tag":143,"props":4705,"children":4706},{},[4707,4715,4720],{"type":45,"tag":170,"props":4708,"children":4709},{},[4710],{"type":45,"tag":76,"props":4711,"children":4712},{},[4713],{"type":51,"value":4714},"Unaligned access",{"type":45,"tag":170,"props":4716,"children":4717},{},[4718],{"type":51,"value":4719},"SEGFAULT or wrong results",{"type":45,"tag":170,"props":4721,"children":4722},{},[4723,4725,4730,4732],{"type":51,"value":4724},"Use ",{"type":45,"tag":60,"props":4726,"children":4728},{"className":4727},[],[4729],{"type":51,"value":4560},{"type":51,"value":4731}," instead of ",{"type":45,"tag":60,"props":4733,"children":4735},{"className":4734},[],[4736],{"type":51,"value":4568},{"type":45,"tag":143,"props":4738,"children":4739},{},[4740,4748,4753],{"type":45,"tag":170,"props":4741,"children":4742},{},[4743],{"type":45,"tag":76,"props":4744,"children":4745},{},[4746],{"type":51,"value":4747},"Missing tail handling",{"type":45,"tag":170,"props":4749,"children":4750},{},[4751],{"type":51,"value":4752},"Wrong results for non-aligned sizes",{"type":45,"tag":170,"props":4754,"children":4755},{},[4756],{"type":51,"value":4757},"Add scalar loop for remainder elements",{"type":45,"tag":143,"props":4759,"children":4760},{},[4761,4769,4774],{"type":45,"tag":170,"props":4762,"children":4763},{},[4764],{"type":45,"tag":76,"props":4765,"children":4766},{},[4767],{"type":51,"value":4768},"OpenMP on small tensor",{"type":45,"tag":170,"props":4770,"children":4771},{},[4772],{"type":51,"value":4773},"Slower than baseline",{"type":45,"tag":170,"props":4775,"children":4776},{},[4777,4779,4785],{"type":51,"value":4778},"Add ",{"type":45,"tag":60,"props":4780,"children":4782},{"className":4781},[],[4783],{"type":51,"value":4784},"if (num_tokens > threshold)",{"type":51,"value":4786}," guard",{"type":45,"tag":143,"props":4788,"children":4789},{},[4790,4798,4803],{"type":45,"tag":170,"props":4791,"children":4792},{},[4793],{"type":45,"tag":76,"props":4794,"children":4795},{},[4796],{"type":51,"value":4797},"Wrong compiler flags",{"type":45,"tag":170,"props":4799,"children":4800},{},[4801],{"type":51,"value":4802},"Intrinsics not recognized",{"type":45,"tag":170,"props":4804,"children":4805},{},[4806,4808,4814],{"type":51,"value":4807},"Check build.toml ",{"type":45,"tag":60,"props":4809,"children":4811},{"className":4810},[],[4812],{"type":51,"value":4813},"cxx-flags",{"type":51,"value":4815}," matches code",{"type":45,"tag":143,"props":4817,"children":4818},{},[4819,4833,4867],{"type":45,"tag":170,"props":4820,"children":4821},{},[4822],{"type":45,"tag":76,"props":4823,"children":4824},{},[4825,4827],{"type":51,"value":4826},"Silent scalar ",{"type":45,"tag":60,"props":4828,"children":4830},{"className":4829},[],[4831],{"type":51,"value":4832},"at::vec",{"type":45,"tag":170,"props":4834,"children":4835},{},[4836,4838,4844,4846,4852,4853,4859,4861],{"type":51,"value":4837},"Kernel ~2x slow, no error; ",{"type":45,"tag":60,"props":4839,"children":4841},{"className":4840},[],[4842],{"type":51,"value":4843},"objdump",{"type":51,"value":4845}," shows 0 ",{"type":45,"tag":60,"props":4847,"children":4849},{"className":4848},[],[4850],{"type":51,"value":4851},"%zmm",{"type":51,"value":4175},{"type":45,"tag":60,"props":4854,"children":4856},{"className":4855},[],[4857],{"type":51,"value":4858},"nm",{"type":51,"value":4860}," shows ",{"type":45,"tag":60,"props":4862,"children":4864},{"className":4863},[],[4865],{"type":51,"value":4866},"expf@GLIBC",{"type":45,"tag":170,"props":4868,"children":4869},{},[4870,4872,4878,4880,4886],{"type":51,"value":4871},"Define ",{"type":45,"tag":60,"props":4873,"children":4875},{"className":4874},[],[4876],{"type":51,"value":4877},"CPU_CAPABILITY_AVX512",{"type":51,"value":4879}," for TUs using ",{"type":45,"tag":60,"props":4881,"children":4883},{"className":4882},[],[4884],{"type":51,"value":4885},"at::vec::Vectorized",{"type":51,"value":4887}," (see build_system.yaml)",{"type":45,"tag":143,"props":4889,"children":4890},{},[4891,4899,4904],{"type":45,"tag":170,"props":4892,"children":4893},{},[4894],{"type":45,"tag":76,"props":4895,"children":4896},{},[4897],{"type":51,"value":4898},"CPUID detection wrong",{"type":45,"tag":170,"props":4900,"children":4901},{},[4902],{"type":51,"value":4903},"Crashes on older CPU",{"type":45,"tag":170,"props":4905,"children":4906},{},[4907,4909,4914],{"type":51,"value":4908},"Verify ",{"type":45,"tag":60,"props":4910,"children":4912},{"className":4911},[],[4913],{"type":51,"value":1374},{"type":51,"value":4915}," checks OS support (XCR0)",{"type":45,"tag":116,"props":4917,"children":4919},{"id":4918},"project-structure",[4920],{"type":51,"value":4921},"Project Structure",{"type":45,"tag":543,"props":4923,"children":4926},{"className":4924,"code":4925,"language":51},[4306],"cpu-kernels\u002F\n├── SKILL.md                                    # This file (skill definition + workflow)\n├── manifest.txt                                # Files included in this skill\n│\n├── scripts\u002F                                    # Standalone CLI tools\n│   ├── analyze_op.py                           # PyTorch op → compute\u002Fmemory analysis\n│   ├── validate_cpu_kernel.py                  # Static checks on C++ kernel code\n│   ├── benchmark_cpu.py                        # Correctness + performance via torch.utils.benchmark\n│   ├── cpu_profiler.py                         # perf stat hardware counters + recommendations\n│   ├── trial_manager.py                        # Tree-structured trial management\n│   ├── config.yaml                             # Session config (max_trials, profiler, build)\n│   └── config.py                               # Shared configuration loader\n│\n└── references\u002F                                 # Knowledge base\n    ├── correctness.yaml                        # Critical constraints for CPU kernels\n    ├── runtime_dispatch.yaml                   # cpu_features.hpp + dispatch pattern\n    ├── build_system.yaml                       # build.toml multi-target CPU compilation\n    ├── simd_optimization_patterns.yaml         # AVX2\u002FAVX512 vector abstractions and patterns\n    ├── quantized_gemm_patterns.yaml            # LUT + tinygemm\u002Fbrgemm template\n    ├── brgemm_patterns.yaml                     # brgemm API, VNNI packing, tinygemm fallback (GEMM kernels only)\n    ├── memory_patterns.yaml                    # Prefetch, alignment, cache blocking\n    ├── threading_patterns.yaml                 # OpenMP parallel patterns\n    ├── dtype_optimizations.yaml                # bf16\u002Ffp8\u002Fint8 handling on CPU\n    ├── optimization_levels.yaml                # Progressive L1→L5 optimization checklist\n    ├── implementation_reference.md             # C++ kernel templates and examples\n    ├── optimization_strategies.md              # Strategy reference + decision tree\n    ├── workflow_details.md                     # Detailed workflow reference\n    └── huggingface-kernels-integration.md      # HF kernels ecosystem integration guide\n",[4927],{"type":45,"tag":60,"props":4928,"children":4929},{"__ignoreMap":548},[4930],{"type":51,"value":4925},{"type":45,"tag":116,"props":4932,"children":4934},{"id":4933},"see-also",[4935],{"type":51,"value":4936},"See Also",{"type":45,"tag":510,"props":4938,"children":4940},{"id":4939},"tools",[4941],{"type":51,"value":4942},"Tools",{"type":45,"tag":1104,"props":4944,"children":4945},{},[4946,4956,4967,4976,4986],{"type":45,"tag":1108,"props":4947,"children":4948},{},[4949,4954],{"type":45,"tag":1071,"props":4950,"children":4952},{"href":4951},"scripts\u002Fanalyze_op.py",[4953],{"type":51,"value":439},{"type":51,"value":4955}," — Analyze PyTorch op characteristics",{"type":45,"tag":1108,"props":4957,"children":4958},{},[4959,4965],{"type":45,"tag":1071,"props":4960,"children":4962},{"href":4961},"scripts\u002Fvalidate_cpu_kernel.py",[4963],{"type":51,"value":4964},"validate_cpu_kernel.py",{"type":51,"value":4966}," — Static kernel validation",{"type":45,"tag":1108,"props":4968,"children":4969},{},[4970,4974],{"type":45,"tag":1071,"props":4971,"children":4972},{"href":1343},[4973],{"type":51,"value":469},{"type":51,"value":4975}," — Correctness + performance measurement",{"type":45,"tag":1108,"props":4977,"children":4978},{},[4979,4984],{"type":45,"tag":1071,"props":4980,"children":4982},{"href":4981},"scripts\u002Fcpu_profiler.py",[4983],{"type":51,"value":476},{"type":51,"value":4985}," — perf stat hardware counters",{"type":45,"tag":1108,"props":4987,"children":4988},{},[4989,4994],{"type":45,"tag":1071,"props":4990,"children":4992},{"href":4991},"scripts\u002Ftrial_manager.py",[4993],{"type":51,"value":96},{"type":51,"value":4995}," — Trial tree management",{"type":45,"tag":510,"props":4997,"children":4999},{"id":4998},"cpu-optimization-references",[5000],{"type":51,"value":5001},"CPU Optimization References",{"type":45,"tag":1104,"props":5003,"children":5004},{},[5005,5014,5023,5032,5041],{"type":45,"tag":1108,"props":5006,"children":5007},{},[5008,5012],{"type":45,"tag":1071,"props":5009,"children":5010},{"href":3346},[5011],{"type":51,"value":4670},{"type":51,"value":5013}," — Critical constraints",{"type":45,"tag":1108,"props":5015,"children":5016},{},[5017,5021],{"type":45,"tag":1071,"props":5018,"children":5019},{"href":3389},[5020],{"type":51,"value":4291},{"type":51,"value":5022}," — SIMD patterns",{"type":45,"tag":1108,"props":5024,"children":5025},{},[5026,5030],{"type":45,"tag":1071,"props":5027,"children":5028},{"href":3406},[5029],{"type":51,"value":4529},{"type":51,"value":5031}," — Quantized GEMM template",{"type":45,"tag":1108,"props":5033,"children":5034},{},[5035,5039],{"type":45,"tag":1071,"props":5036,"children":5037},{"href":2736},[5038],{"type":51,"value":2739},{"type":51,"value":5040}," — Progressive optimization",{"type":45,"tag":1108,"props":5042,"children":5043},{},[5044,5049],{"type":45,"tag":1071,"props":5045,"children":5046},{"href":3329},[5047],{"type":51,"value":5048},"implementation_reference.md",{"type":51,"value":5050}," — Code templates",{"type":45,"tag":510,"props":5052,"children":5054},{"id":5053},"external-resources",[5055],{"type":51,"value":5056},"External Resources",{"type":45,"tag":1104,"props":5058,"children":5059},{},[5060,5072,5082,5092,5103],{"type":45,"tag":1108,"props":5061,"children":5062},{},[5063,5070],{"type":45,"tag":1071,"props":5064,"children":5067},{"href":25,"rel":5065},[5066],"nofollow",[5068],{"type":51,"value":5069},"Hugging Face Kernels",{"type":51,"value":5071}," — Kernel hub and builder CLI",{"type":45,"tag":1108,"props":5073,"children":5074},{},[5075],{"type":45,"tag":1071,"props":5076,"children":5079},{"href":5077,"rel":5078},"https:\u002F\u002Fwww.intel.com\u002Fcontent\u002Fwww\u002Fus\u002Fen\u002Fdocs\u002Fintrinsics-guide\u002F",[5066],[5080],{"type":51,"value":5081},"Intel Intrinsics Guide",{"type":45,"tag":1108,"props":5083,"children":5084},{},[5085],{"type":45,"tag":1071,"props":5086,"children":5089},{"href":5087,"rel":5088},"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fkernels\u002Ftree\u002Fmain\u002Fdocs",[5066],[5090],{"type":51,"value":5091},"kernel-builder Documentation",{"type":45,"tag":1108,"props":5093,"children":5094},{},[5095,5101],{"type":45,"tag":1071,"props":5096,"children":5098},{"href":5097},"..\u002Fxpu-kernels\u002FSKILL.md",[5099],{"type":51,"value":5100},"xpu-kernels skill",{"type":51,"value":5102}," — the Intel XPU Triton skill this workflow was adapted from",{"type":45,"tag":1108,"props":5104,"children":5105},{},[5106,5113],{"type":45,"tag":1071,"props":5107,"children":5110},{"href":5108,"rel":5109},"https:\u002F\u002Fgithub.com\u002FIntelLabs\u002FXe-Forge",[5066],[5111],{"type":51,"value":5112},"Xe-Forge",{"type":51,"value":5114}," — the LLM-driven optimization framework the skill methodology originates from",{"type":45,"tag":116,"props":5116,"children":5118},{"id":5117},"acknowledgments",[5119],{"type":51,"value":5120},"Acknowledgments",{"type":45,"tag":54,"props":5122,"children":5123},{},[5124,5126,5130,5132,5137],{"type":51,"value":5125},"The methodology of this skill — the YAML knowledge base, the benchmark\u002Fvalidation harnesses, and the branching trial-manager optimization loop — was adapted from the ",{"type":45,"tag":1071,"props":5127,"children":5128},{"href":5097},[5129],{"type":51,"value":5100},{"type":51,"value":5131}," built by a group of Intel AI researchers, the IntelLabs team behind ",{"type":45,"tag":1071,"props":5133,"children":5135},{"href":5108,"rel":5134},[5066],[5136],{"type":51,"value":5112},{"type":51,"value":5138},", where the methodology originates. Thanks to the original authors for a solid foundation to build on.",{"type":45,"tag":5140,"props":5141,"children":5142},"style",{},[5143],{"type":51,"value":5144},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5146,"total":632},[5147,5154,5171,5187],{"slug":4,"name":4,"fn":5,"description":6,"org":5148,"tags":5149,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5150,5151,5152,5153],{"name":19,"slug":20,"type":15},{"name":22,"slug":23,"type":15},{"name":9,"slug":17,"type":15},{"name":13,"slug":14,"type":15},{"slug":5155,"name":5155,"fn":5156,"description":5157,"org":5158,"tags":5159,"stars":24,"repoUrl":25,"updatedAt":5170},"cuda-kernels","write and benchmark optimized CUDA kernels","Provides guidance for writing and benchmarking optimized CUDA kernels for NVIDIA GPUs (H100, A100, T4) targeting HuggingFace diffusers and transformers libraries. Kernels must be kernel-builder\u002FABI3-compliant: no pybind11, no setup.py, TORCH_LIBRARY_EXPAND bindings only. Supports models like LTX-Video, Stable Diffusion, LLaMA, Mistral, and Qwen. Includes integration with HuggingFace Kernels Hub (get_kernel) for loading pre-compiled kernels. Includes benchmarking scripts to compare kernel performance against baseline implementations.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5160,5163,5166,5167,5168],{"name":5161,"slug":5162,"type":15},"AI Infrastructure","ai-infrastructure",{"name":5164,"slug":5165,"type":15},"Deep Learning","deep-learning",{"name":9,"slug":17,"type":15},{"name":13,"slug":14,"type":15},{"name":5169,"slug":583,"type":15},"Python","2026-05-15T06:18:29.266429",{"slug":5172,"name":5172,"fn":5173,"description":5174,"org":5175,"tags":5176,"stars":24,"repoUrl":25,"updatedAt":5186},"rocm-kernels","build optimized Triton kernels for AMD GPUs","Provides guidance for writing and benchmarking optimized Triton kernels for AMD GPUs (MI355X, R9700) on ROCm, targeting HuggingFace diffusers (LTX-Video, SD3, FLUX) and transformers. Core kernels: RMSNorm, RoPE 3D, GEGLU, AdaLN. Includes XCD swizzle, autotune, diffusers integration patterns, and LTX-Video pipeline injection.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5177,5178,5179,5180,5183],{"name":5161,"slug":5162,"type":15},{"name":5164,"slug":5165,"type":15},{"name":13,"slug":14,"type":15},{"name":5181,"slug":5182,"type":15},"ROCm","rocm",{"name":5184,"slug":5185,"type":15},"Triton","triton","2026-04-16T05:06:54.179351",{"slug":5188,"name":5188,"fn":5189,"description":5190,"org":5191,"tags":5192,"stars":24,"repoUrl":25,"updatedAt":5197},"xpu-kernels","optimize Triton kernels for Intel XPU","Provides guidance for writing, optimizing, and benchmarking Triton kernels for Intel XPU GPUs (Battlemage\u002FArc Pro B50) using the Xe-Forge optimization framework. Includes an LLM-driven trial-loop workflow (analyze, validate, benchmark, profile, finalize), XPU-specific patterns (tensor descriptors, GRF mode, tile swizzling), KernelBench fused kernels, and Flash Attention.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5193,5194,5195,5196],{"name":5161,"slug":5162,"type":15},{"name":5164,"slug":5165,"type":15},{"name":13,"slug":14,"type":15},{"name":5184,"slug":5185,"type":15},"2026-07-18T05:15:18.767865",{"items":5199,"total":5366},[5200,5218,5232,5249,5263,5274,5287,5302,5316,5326,5339,5351],{"slug":5201,"name":5201,"fn":5202,"description":5203,"org":5204,"tags":5205,"stars":5215,"repoUrl":5216,"updatedAt":5217},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5206,5207,5208,5211,5212],{"name":5164,"slug":5165,"type":15},{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},"LLM","llm",{"name":5169,"slug":583,"type":15},{"name":5213,"slug":5214,"type":15},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":5219,"name":5219,"fn":5220,"description":5221,"org":5222,"tags":5223,"stars":5229,"repoUrl":5230,"updatedAt":5231},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5224,5225,5226,5227,5228],{"name":5161,"slug":5162,"type":15},{"name":5164,"slug":5165,"type":15},{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},{"name":5169,"slug":583,"type":15},18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828",{"slug":5233,"name":5233,"fn":5234,"description":5235,"org":5236,"tags":5237,"stars":5246,"repoUrl":5247,"updatedAt":5248},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5238,5241,5244,5245],{"name":5239,"slug":5240,"type":15},"CLI","cli",{"name":5242,"slug":5243,"type":15},"Datasets","datasets",{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-04-06T18:25:34.020855",{"slug":5250,"name":5250,"fn":5251,"description":5252,"org":5253,"tags":5254,"stars":5246,"repoUrl":5247,"updatedAt":5262},"hf-cloud-aws-context-discovery","discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5255,5258,5259],{"name":5256,"slug":5257,"type":15},"AWS","aws",{"name":5239,"slug":5240,"type":15},{"name":5260,"slug":5261,"type":15},"Configuration","configuration","2026-07-08T05:55:33.716099",{"slug":5264,"name":5264,"fn":5265,"description":5266,"org":5267,"tags":5268,"stars":5246,"repoUrl":5247,"updatedAt":5273},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5269,5270,5271,5272],{"name":5161,"slug":5162,"type":15},{"name":5256,"slug":5257,"type":15},{"name":22,"slug":23,"type":15},{"name":5169,"slug":583,"type":15},"2026-07-08T05:55:32.505017",{"slug":5275,"name":5275,"fn":5276,"description":5277,"org":5278,"tags":5279,"stars":5246,"repoUrl":5247,"updatedAt":5286},"hf-cloud-sagemaker-deployment-planner","plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5280,5281,5282,5285],{"name":5161,"slug":5162,"type":15},{"name":5256,"slug":5257,"type":15},{"name":5283,"slug":5284,"type":15},"Deployment","deployment",{"name":9,"slug":17,"type":15},"2026-07-08T05:55:37.387689",{"slug":5288,"name":5288,"fn":5289,"description":5290,"org":5291,"tags":5292,"stars":5246,"repoUrl":5247,"updatedAt":5301},"hf-cloud-sagemaker-iam-preflight","configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5293,5294,5295,5298],{"name":5161,"slug":5162,"type":15},{"name":5256,"slug":5257,"type":15},{"name":5296,"slug":5297,"type":15},"Permissions","permissions",{"name":5299,"slug":5300,"type":15},"Security","security","2026-07-08T05:55:34.948771",{"slug":5303,"name":5303,"fn":5304,"description":5305,"org":5306,"tags":5307,"stars":5246,"repoUrl":5247,"updatedAt":5315},"hf-cloud-sagemaker-production-defaults","create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5308,5309,5310,5311,5312],{"name":5161,"slug":5162,"type":15},{"name":5256,"slug":5257,"type":15},{"name":5283,"slug":5284,"type":15},{"name":9,"slug":17,"type":15},{"name":5313,"slug":5314,"type":15},"Monitoring","monitoring","2026-07-08T05:55:38.664702",{"slug":5317,"name":5317,"fn":5318,"description":5319,"org":5320,"tags":5321,"stars":5246,"repoUrl":5247,"updatedAt":5325},"hf-cloud-serving-image-selection","select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5322,5323,5324],{"name":5161,"slug":5162,"type":15},{"name":5256,"slug":5257,"type":15},{"name":5283,"slug":5284,"type":15},"2026-07-08T05:55:36.173465",{"slug":5327,"name":5327,"fn":5328,"description":5329,"org":5330,"tags":5331,"stars":5246,"repoUrl":5247,"updatedAt":5338},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5332,5333,5334,5335],{"name":5242,"slug":5243,"type":15},{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},{"name":5336,"slug":5337,"type":15},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":5340,"name":5340,"fn":5341,"description":5342,"org":5343,"tags":5344,"stars":5246,"repoUrl":5247,"updatedAt":5350},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5345,5346,5347,5348,5349],{"name":5161,"slug":5162,"type":15},{"name":5239,"slug":5240,"type":15},{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},{"name":13,"slug":14,"type":15},"2026-06-13T07:23:57.101435",{"slug":5352,"name":5352,"fn":5353,"description":5354,"org":5355,"tags":5356,"stars":5246,"repoUrl":5247,"updatedAt":5365},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[5357,5360,5361,5362],{"name":5358,"slug":5359,"type":15},"Analytics","analytics",{"name":9,"slug":17,"type":15},{"name":5209,"slug":5210,"type":15},{"name":5363,"slug":5364,"type":15},"Research","research","2026-04-24T05:09:45.870658",37]