[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-cupynumeric-hdf5":3,"mdc--wwvjb5-key":31,"related-repo-nvidia-cupynumeric-hdf5":1992,"related-org-nvidia-cupynumeric-hdf5":2096},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":29,"mdContent":30},"cupynumeric-hdf5","read and write cuPyNumeric arrays to HDF5","Read and write large cuPyNumeric arrays to HDF5 with Legate's parallel, distributed HDF5 I\u002FO (legate.io.hdf5: to_file, from_file, from_file_batched). Use when a developer needs to save a cuPyNumeric array to an .h5\u002F.hdf5 file, load an HDF5 dataset into a distributed cuPyNumeric array, read a large HDF5 dataset in chunks, hand arrays to an HPC pipeline as a single file, or accelerate HDF5 disk I\u002FO with GPUDirect Storage (GDS). Do not use it for Parquet\u002FcuDF\u002Fraw-binary or other sharded\u002Fcustom layouts (see the cupynumeric-parallel-data-load skill), Zarr or object-store\u002FS3 output, .npz or pickled archives, plain h5py without cuPyNumeric, or pure array compute such as FFT, matmul, or reductions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17],{"name":13,"slug":14,"type":15},"Data Engineering","data-engineering","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Python","python",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:30:17.861772","CC-BY-4.0 OR Apache-2.0",281,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":28},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fcupynumeric-hdf5","---\nname: cupynumeric-hdf5\ndescription: >-\n  Read and write large cuPyNumeric arrays to HDF5 with Legate's parallel, distributed HDF5 I\u002FO (legate.io.hdf5: to_file, from_file, from_file_batched). Use when a developer needs to save a cuPyNumeric array to an .h5\u002F.hdf5 file, load an HDF5 dataset into a distributed cuPyNumeric array, read a large HDF5 dataset in chunks, hand arrays to an HPC pipeline as a single file, or accelerate HDF5 disk I\u002FO with GPUDirect Storage (GDS). Do not use it for Parquet\u002FcuDF\u002Fraw-binary or other sharded\u002Fcustom layouts (see the cupynumeric-parallel-data-load skill), Zarr or object-store\u002FS3 output, .npz or pickled archives, plain h5py without cuPyNumeric, or pure array compute such as FFT, matmul, or reductions.\nlicense: CC-BY-4.0 OR Apache-2.0\ncompatibility: >-\n  Requires cuPyNumeric and Legate 26.01 or newer (the legate.io.hdf5 module; in 25.03 it lived at legate.core.io.hdf5). Requires h5py (conda install -c conda-forge h5py) - hdf5.py imports it at module load, so the import fails without it. GPUDirect Storage is optional and needs the nv-legate vfd-gds plugin (bundled with legate) plus NVIDIA cuFile.\nmetadata:\n  version: \"2.0.0\"\n  author: \"NVIDIA Corporation \u003Clegate@nvidia.com>\"\n  tags:\n  - hdf5\n  - cupynumeric\n  - legate\n  - data-io\n  - h5py\n  - gpudirect-storage\n  - parallel-io\n  - scientific-data\n  upstream: https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fcupynumeric\n  docs: https:\u002F\u002Fdocs.nvidia.com\u002Flegate\u002Flatest\u002Fapi\u002Fpython\u002Fio\u002Findex.html\n---\n\n# cuPyNumeric HDF5 I\u002FO\n\n## Purpose\n\nUse [`legate.io.hdf5`](https:\u002F\u002Fdocs.nvidia.com\u002Flegate\u002Flatest\u002Fapi\u002Fpython\u002Fio\u002Findex.html) to read and write [cuPyNumeric](https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fcupynumeric) arrays as [HDF5](https:\u002F\u002Fwww.hdfgroup.org\u002Fsolutions\u002Fhdf5\u002F) files. Reach for it whenever a cuPyNumeric array must land in — or load from — an `.h5`\u002F`.hdf5` file: every rank reads and writes its own tile in parallel, so never funnel a large array through a single process.\n\n**Answer inline.** Treat the snippets and rules below as complete and verified — answer save \u002F load \u002F stream \u002F fence \u002F bridge questions directly, without opening the `assets\u002F` scripts or reading the installed `legate` source. Reach for the assets only to *run* a verification.\n\n## Activate\n\nActivate when the user asks about: saving a cuPyNumeric array to an `.h5` \u002F `.hdf5` file, loading an HDF5 dataset into a cuPyNumeric array, reading a large HDF5 dataset in chunks, producing a single file for an HPC post-processing pipeline, or speeding up HDF5 disk I\u002FO with GPUDirect Storage.\n\n## When NOT to use\n\nRedirect these requests elsewhere instead of reaching for `legate.io.hdf5`:\n\n- **Route Parquet \u002F Arrow \u002F cuDF, raw-binary, or sharded \u002F custom on-disk layouts to the cupynumeric-parallel-data-load skill** — it owns cuPyNumeric's no-built-in-loader paths; `legate.io.hdf5` covers single-file HDF5 only.\n- **Answer pure array compute with cuPyNumeric ops** (FFT, matmul, reductions, slicing, linear algebra) — this skill covers disk I\u002FO only.\n- **Send chunked or object-store (S3) output to a chunked format such as Zarr** — not single-file HDF5.\n- **Load `.npz` or pickled archives with NumPy** (`np.load`), then bridge with `cn.asarray(...)` — `legate.io.hdf5` reads HDF5 only, and `cupynumeric.load` reads single `.npy` only.\n- **Use h5py directly for plain HDF5 reads with no cuPyNumeric\u002FLegate** — `with h5py.File(path, \"r\") as f: arr = f[\"dataset\"][:]`.\n\n## Prerequisites\n\nInstall h5py before importing anything from `legate.io.hdf5`:\n\n```bash\nconda install -c conda-forge h5py        # required; legate\u002Fio\u002Fhdf5.py imports it at load\n```\n\nExpect `from legate.io.hdf5 import ...` to raise `ModuleNotFoundError` until you do — the module imports `h5py` at load time. ([h5py](https:\u002F\u002Fwww.h5py.org\u002F) · [conda-forge build](https:\u002F\u002Fanaconda.org\u002Fconda-forge\u002Fh5py))\n\n## API\n\n| Function | Signature | Purpose |\n|---|---|---|\n| `to_file` | `to_file(array, path, dataset_name)` | Write a cuPyNumeric array \u002F `LogicalArray` to one HDF5 file as a virtual dataset (VDS) — each rank writes its own tile. |\n| `from_file` | `from_file(path, dataset_name) -> LogicalArray` | Read one HDF5 dataset into a distributed array. |\n| `from_file_batched` | `from_file_batched(path, dataset_name, chunk_size) -> Iterator[(LogicalArray, offsets)]` | Read a dataset in chunks — chunks the file read, not the assembled array. |\n\nImport all three from `legate.io.hdf5`. Always pass `dataset_name` as the full path to a single array inside the file (e.g. `\"\u002Fdata\"` or `\"\u002Fgroup\u002Fx\"`), never a group.\n\n## Examples\n\n### Round trip\n\n```python\nimport cupynumeric as cn\nfrom legate.core import get_legate_runtime\nfrom legate.io.hdf5 import from_file, to_file\n\na = cn.arange(64, dtype=cn.float32).reshape(8, 8)\n\n# Write: pass the cuPyNumeric ndarray straight in - no manual conversion.\nto_file(array=a, path=\"out.h5\", dataset_name=\"\u002Fdata\")\nget_legate_runtime().issue_execution_fence(block=True)   # needed before any external reader\n\n# Read: from_file returns a legate LogicalArray; cn.asarray bridges it back.\nb = cn.asarray(from_file(\"out.h5\", dataset_name=\"\u002Fdata\"))\nassert cn.array_equal(a, b)\n```\n\nRun `assets\u002Fhdf5_roundtrip.py` to verify (optional — not needed to answer).\n\n### Read a large file in chunks\n\nUse `from_file_batched` to read the source file in chunks instead of pulling it into host memory all at once. It yields one `LogicalArray` per chunk plus that chunk's offsets in the global shape. Expect clipped boundary chunks (an axis of length 5 with `chunk_size=2` yields 2, 2, 1), so place each chunk by its actual shape, not the requested `chunk_size`. Note that this chunks the *file read*, not the result — the assembled array (`out`) still has to fit in distributed memory:\n\n```python\nimport h5py\nimport cupynumeric as cn\nfrom legate.core import get_legate_runtime\nfrom legate.io.hdf5 import from_file_batched\n\nwith h5py.File(\"big.h5\", \"r\") as f:          # read shape\u002Fdtype without loading data\n    shape, dtype = f[\"data\"].shape, f[\"data\"].dtype\n\nout = cn.empty(shape, dtype=dtype)\nfor chunk, (r0, c0) in from_file_batched(\"big.h5\", \"data\", chunk_size=(4096, 4096)):\n    out[r0:r0 + chunk.shape[0], c0:c0 + chunk.shape[1]] = cn.asarray(chunk)\nget_legate_runtime().issue_execution_fence(block=True)\n```\n\nKeep every `chunk_size` entry positive and its length equal to the dataset's rank, or `from_file_batched` raises `ValueError`. Run `assets\u002Fhdf5_batched_read.py` to verify (optional).\n\n## Instructions\n\n- **Pass the cuPyNumeric ndarray directly to `to_file`** - it implements `__legate_data_interface__`, which `to_file` accepts as `LogicalArrayLike`. Skip any `np.array(...)` round-trip.\n- **Bridge results back with `cn.asarray(...)`.** `from_file` and each `from_file_batched` chunk return a Legate `LogicalArray`; wrap it with `cn.asarray(la)` to get a cuPyNumeric ndarray (zero-copy, no host bounce).\n- **Fence before any external reader.** Legate I\u002FO is asynchronous: `to_file` only queues the write. Insert `get_legate_runtime().issue_execution_fence(block=True)` before h5py, a subprocess, or another tool opens the file. Skip the fence for a `from_file`\n  issued later in the same Legate program — the runtime preserves that ordering.\n- **Run from outside the cuPyNumeric source tree** (e.g. `cd \u002Ftmp`). Python puts the cwd first on `sys.path`, so an in-tree `cupynumeric\u002F` directory shadows the installed package (`ModuleNotFoundError: cupynumeric.install_info`).\n- **Give every rank the same `path`.** The program runs on every rank (SPMD), so pass `to_file`\u002F`from_file` an identical `path` on each — a per-rank `tempfile.mkstemp()` name breaks the collective I\u002FO. When the program creates the file itself, write it with the collective `to_file`, not a per-rank `h5py` write.\n\n## `to_file` behavior to plan around\n\n- Expect an HDF5 **virtual dataset (VDS)**: each rank writes its own tile and the file presents them as one logical dataset.\n- Treat `to_file` as **destructive** — it overwrites `path` if it already exists, so guard any file you must not clobber.\n- Let `to_file` **create missing parent directories**; do not pre-create them.\n- Give `path` a file name (`\u002Fpath\u002Fto\u002Ffile.h5`), never a directory — a directory raises `ValueError`. Pass a **bound** array (one with a known shape); `to_file` raises `ValueError` on an *unbound* array — a Legate array created without a shape (e.g. `create_array(dtype, ndim=n)`) whose extent a producing task fills in later. cuPyNumeric ndarrays are always bound — even lazy\u002Fdeferred ones — so this only affects raw `LogicalArray`s.\n\n## GPUDirect Storage (GDS)\n\n**Always set `LEGATE_IO_USE_VFD_GDS=1` for runs that read HDF5 into GPU memory** — whether or not the cluster has GPUDirect-capable storage:\n\n```bash\nexport LEGATE_IO_USE_VFD_GDS=1          # set before launching\n# or, with the legate driver:\nlegate --io-use-vfd-gds my_script.py\n```\n\n- **Read into the GPU through the GDS VFD, not the default path.** The default (POSIX) VFD stages each GPU read through zero-copy memory (ZCMEM), of which Legate reserves only 128 MB — so a GPU read of an array larger than ~128 MB aborts. The GDS VFD removes that staging buffer.\n- **Leave it unset when reading into host (CPU) memory** — the VFD GDS plugin is unnecessary there and only adds overhead.\n- **Keep `=1` even without GPUDirect-capable storage** — cuFile falls back to compatibility mode automatically (set `export CUFILE_ALLOW_COMPAT_MODE=true` if it is not already on), and `=1` still avoids the ZCMEM abort.\n- **Attribute it correctly:** the GDS VFD is the [nv-legate\u002Fvfd-gds](https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fvfd-gds) plugin over NVIDIA [cuFile](https:\u002F\u002Fdeveloper.nvidia.com\u002Fgpudirect-storage), **not** KvikIO (KvikIO backs Legate's Zarr\u002Ftile I\u002FO, not HDF5). Confirm it engaged by grepping the run log for `H5FD__gds_open: Successfully opened file w\u002FGDS VFD`.\n\n## Troubleshooting\n\n| Symptom | Cause and fix |\n|---|---|\n| `ModuleNotFoundError: No module named 'h5py'` on import | h5py is missing — `conda install -c conda-forge h5py`. |\n| File looks empty\u002Ftruncated to h5py right after `to_file` | The async write hasn't landed — add `get_legate_runtime().issue_execution_fence(block=True)` before the external read. |\n| `ValueError` from `to_file` | `path` is a directory — pass a file path such as `results\u002Fdata.h5`. |\n| `ModuleNotFoundError: No module named 'cupynumeric.install_info'` | Running inside the source tree — `cd \u002Ftmp` (any directory outside the repo). |\n| Abort\u002Fcrash reading a GPU array ≳128 MB | Default 128 MB ZCMEM staging buffer — set `LEGATE_IO_USE_VFD_GDS=1` for GPU reads. |\n| `from_file` returned `LogicalArray(...)` | Expected — wrap it with `cn.asarray(...)`. |\n\n## Limitations & version notes\n\n- **Import from `legate.io.hdf5`** (Legate 26.01+); rewrite any `legate.core.io.hdf5` import left over from the 25.03 line (e.g. the [25.03 launch blog](https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fnvidia-cupynumeric-25-03-now-fully-open-source-with-pip-and-hdf5-support\u002F) still shows the old path).\n- **Install h5py explicitly** — it ships in no default cuPyNumeric env.\n- **Point `dataset_name` at a single array, never a group**; traverse groups with h5py first to discover dataset paths.\n- **On GPU, always read with `LEGATE_IO_USE_VFD_GDS=1`** (see [GPUDirect Storage](#gpudirect-storage-gds)) — the default path aborts on GPU arrays larger than the 128 MB ZCMEM buffer. Leave it unset for CPU reads.\n\n## Verify\n\n```bash\ncd \u002Ftmp                                  # outside the cupynumeric source tree\nconda install -c conda-forge h5py        # one-time, if not already present\nLEGATE_CONFIG=\"--cpus 4\" LEGATE_AUTO_CONFIG=0 python \u003Cskill>\u002Fassets\u002Fhdf5_roundtrip.py\nLEGATE_CONFIG=\"--cpus 4\" LEGATE_AUTO_CONFIG=0 python \u003Cskill>\u002Fassets\u002Fhdf5_batched_read.py\n```\n\nExpect `HDF5 ROUND TRIP OK` and `HDF5 BATCHED READ OK`. Add `--gpus 1` (and `LEGATE_IO_USE_VFD_GDS=1`) to exercise the GPU \u002F GDS path.\n",{"data":32,"body":48},{"name":4,"description":6,"license":23,"compatibility":33,"metadata":34},"Requires cuPyNumeric and Legate 26.01 or newer (the legate.io.hdf5 module; in 25.03 it lived at legate.core.io.hdf5). Requires h5py (conda install -c conda-forge h5py) - hdf5.py imports it at module load, so the import fails without it. GPUDirect Storage is optional and needs the nv-legate vfd-gds plugin (bundled with legate) plus NVIDIA cuFile.",{"version":35,"author":36,"tags":37,"upstream":46,"docs":47},"2.0.0","NVIDIA Corporation \u003Clegate@nvidia.com>",[38,39,40,41,42,43,44,45],"hdf5","cupynumeric","legate","data-io","h5py","gpudirect-storage","parallel-io","scientific-data","https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fcupynumeric","https:\u002F\u002Fdocs.nvidia.com\u002Flegate\u002Flatest\u002Fapi\u002Fpython\u002Fio\u002Findex.html",{"type":49,"children":50},"root",[51,60,67,121,155,161,180,186,198,314,320,331,381,426,432,550,586,592,599,722,735,741,790,889,924,930,1157,1168,1296,1302,1320,1384,1482,1488,1670,1676,1761,1767,1951,1986],{"type":52,"tag":53,"props":54,"children":56},"element","h1",{"id":55},"cupynumeric-hdf5-io",[57],{"type":58,"value":59},"text","cuPyNumeric HDF5 I\u002FO",{"type":52,"tag":61,"props":62,"children":64},"h2",{"id":63},"purpose",[65],{"type":58,"value":66},"Purpose",{"type":52,"tag":68,"props":69,"children":70},"p",{},[71,73,86,88,94,96,103,105,111,113,119],{"type":58,"value":72},"Use ",{"type":52,"tag":74,"props":75,"children":78},"a",{"href":47,"rel":76},[77],"nofollow",[79],{"type":52,"tag":80,"props":81,"children":83},"code",{"className":82},[],[84],{"type":58,"value":85},"legate.io.hdf5",{"type":58,"value":87}," to read and write ",{"type":52,"tag":74,"props":89,"children":91},{"href":46,"rel":90},[77],[92],{"type":58,"value":93},"cuPyNumeric",{"type":58,"value":95}," arrays as ",{"type":52,"tag":74,"props":97,"children":100},{"href":98,"rel":99},"https:\u002F\u002Fwww.hdfgroup.org\u002Fsolutions\u002Fhdf5\u002F",[77],[101],{"type":58,"value":102},"HDF5",{"type":58,"value":104}," files. Reach for it whenever a cuPyNumeric array must land in — or load from — an ",{"type":52,"tag":80,"props":106,"children":108},{"className":107},[],[109],{"type":58,"value":110},".h5",{"type":58,"value":112},"\u002F",{"type":52,"tag":80,"props":114,"children":116},{"className":115},[],[117],{"type":58,"value":118},".hdf5",{"type":58,"value":120}," file: every rank reads and writes its own tile in parallel, so never funnel a large array through a single process.",{"type":52,"tag":68,"props":122,"children":123},{},[124,130,132,138,140,145,147,153],{"type":52,"tag":125,"props":126,"children":127},"strong",{},[128],{"type":58,"value":129},"Answer inline.",{"type":58,"value":131}," Treat the snippets and rules below as complete and verified — answer save \u002F load \u002F stream \u002F fence \u002F bridge questions directly, without opening the ",{"type":52,"tag":80,"props":133,"children":135},{"className":134},[],[136],{"type":58,"value":137},"assets\u002F",{"type":58,"value":139}," scripts or reading the installed ",{"type":52,"tag":80,"props":141,"children":143},{"className":142},[],[144],{"type":58,"value":40},{"type":58,"value":146}," source. Reach for the assets only to ",{"type":52,"tag":148,"props":149,"children":150},"em",{},[151],{"type":58,"value":152},"run",{"type":58,"value":154}," a verification.",{"type":52,"tag":61,"props":156,"children":158},{"id":157},"activate",[159],{"type":58,"value":160},"Activate",{"type":52,"tag":68,"props":162,"children":163},{},[164,166,171,173,178],{"type":58,"value":165},"Activate when the user asks about: saving a cuPyNumeric array to an ",{"type":52,"tag":80,"props":167,"children":169},{"className":168},[],[170],{"type":58,"value":110},{"type":58,"value":172}," \u002F ",{"type":52,"tag":80,"props":174,"children":176},{"className":175},[],[177],{"type":58,"value":118},{"type":58,"value":179}," file, loading an HDF5 dataset into a cuPyNumeric array, reading a large HDF5 dataset in chunks, producing a single file for an HPC post-processing pipeline, or speeding up HDF5 disk I\u002FO with GPUDirect Storage.",{"type":52,"tag":61,"props":181,"children":183},{"id":182},"when-not-to-use",[184],{"type":58,"value":185},"When NOT to use",{"type":52,"tag":68,"props":187,"children":188},{},[189,191,196],{"type":58,"value":190},"Redirect these requests elsewhere instead of reaching for ",{"type":52,"tag":80,"props":192,"children":194},{"className":193},[],[195],{"type":58,"value":85},{"type":58,"value":197},":",{"type":52,"tag":199,"props":200,"children":201},"ul",{},[202,220,230,240,297],{"type":52,"tag":203,"props":204,"children":205},"li",{},[206,211,213,218],{"type":52,"tag":125,"props":207,"children":208},{},[209],{"type":58,"value":210},"Route Parquet \u002F Arrow \u002F cuDF, raw-binary, or sharded \u002F custom on-disk layouts to the cupynumeric-parallel-data-load skill",{"type":58,"value":212}," — it owns cuPyNumeric's no-built-in-loader paths; ",{"type":52,"tag":80,"props":214,"children":216},{"className":215},[],[217],{"type":58,"value":85},{"type":58,"value":219}," covers single-file HDF5 only.",{"type":52,"tag":203,"props":221,"children":222},{},[223,228],{"type":52,"tag":125,"props":224,"children":225},{},[226],{"type":58,"value":227},"Answer pure array compute with cuPyNumeric ops",{"type":58,"value":229}," (FFT, matmul, reductions, slicing, linear algebra) — this skill covers disk I\u002FO only.",{"type":52,"tag":203,"props":231,"children":232},{},[233,238],{"type":52,"tag":125,"props":234,"children":235},{},[236],{"type":58,"value":237},"Send chunked or object-store (S3) output to a chunked format such as Zarr",{"type":58,"value":239}," — not single-file HDF5.",{"type":52,"tag":203,"props":241,"children":242},{},[243,256,258,264,266,272,274,279,281,287,289,295],{"type":52,"tag":125,"props":244,"children":245},{},[246,248,254],{"type":58,"value":247},"Load ",{"type":52,"tag":80,"props":249,"children":251},{"className":250},[],[252],{"type":58,"value":253},".npz",{"type":58,"value":255}," or pickled archives with NumPy",{"type":58,"value":257}," (",{"type":52,"tag":80,"props":259,"children":261},{"className":260},[],[262],{"type":58,"value":263},"np.load",{"type":58,"value":265},"), then bridge with ",{"type":52,"tag":80,"props":267,"children":269},{"className":268},[],[270],{"type":58,"value":271},"cn.asarray(...)",{"type":58,"value":273}," — ",{"type":52,"tag":80,"props":275,"children":277},{"className":276},[],[278],{"type":58,"value":85},{"type":58,"value":280}," reads HDF5 only, and ",{"type":52,"tag":80,"props":282,"children":284},{"className":283},[],[285],{"type":58,"value":286},"cupynumeric.load",{"type":58,"value":288}," reads single ",{"type":52,"tag":80,"props":290,"children":292},{"className":291},[],[293],{"type":58,"value":294},".npy",{"type":58,"value":296}," only.",{"type":52,"tag":203,"props":298,"children":299},{},[300,305,306,312],{"type":52,"tag":125,"props":301,"children":302},{},[303],{"type":58,"value":304},"Use h5py directly for plain HDF5 reads with no cuPyNumeric\u002FLegate",{"type":58,"value":273},{"type":52,"tag":80,"props":307,"children":309},{"className":308},[],[310],{"type":58,"value":311},"with h5py.File(path, \"r\") as f: arr = f[\"dataset\"][:]",{"type":58,"value":313},".",{"type":52,"tag":61,"props":315,"children":317},{"id":316},"prerequisites",[318],{"type":58,"value":319},"Prerequisites",{"type":52,"tag":68,"props":321,"children":322},{},[323,325,330],{"type":58,"value":324},"Install h5py before importing anything from ",{"type":52,"tag":80,"props":326,"children":328},{"className":327},[],[329],{"type":58,"value":85},{"type":58,"value":197},{"type":52,"tag":332,"props":333,"children":338},"pre",{"className":334,"code":335,"language":336,"meta":337,"style":337},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","conda install -c conda-forge h5py        # required; legate\u002Fio\u002Fhdf5.py imports it at load\n","bash","",[339],{"type":52,"tag":80,"props":340,"children":341},{"__ignoreMap":337},[342],{"type":52,"tag":343,"props":344,"children":347},"span",{"class":345,"line":346},"line",1,[348,354,360,365,370,375],{"type":52,"tag":343,"props":349,"children":351},{"style":350},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[352],{"type":58,"value":353},"conda",{"type":52,"tag":343,"props":355,"children":357},{"style":356},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[358],{"type":58,"value":359}," install",{"type":52,"tag":343,"props":361,"children":362},{"style":356},[363],{"type":58,"value":364}," -c",{"type":52,"tag":343,"props":366,"children":367},{"style":356},[368],{"type":58,"value":369}," conda-forge",{"type":52,"tag":343,"props":371,"children":372},{"style":356},[373],{"type":58,"value":374}," h5py",{"type":52,"tag":343,"props":376,"children":378},{"style":377},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[379],{"type":58,"value":380},"        # required; legate\u002Fio\u002Fhdf5.py imports it at load\n",{"type":52,"tag":68,"props":382,"children":383},{},[384,386,392,394,400,402,407,409,415,417,424],{"type":58,"value":385},"Expect ",{"type":52,"tag":80,"props":387,"children":389},{"className":388},[],[390],{"type":58,"value":391},"from legate.io.hdf5 import ...",{"type":58,"value":393}," to raise ",{"type":52,"tag":80,"props":395,"children":397},{"className":396},[],[398],{"type":58,"value":399},"ModuleNotFoundError",{"type":58,"value":401}," until you do — the module imports ",{"type":52,"tag":80,"props":403,"children":405},{"className":404},[],[406],{"type":58,"value":42},{"type":58,"value":408}," at load time. (",{"type":52,"tag":74,"props":410,"children":413},{"href":411,"rel":412},"https:\u002F\u002Fwww.h5py.org\u002F",[77],[414],{"type":58,"value":42},{"type":58,"value":416}," · ",{"type":52,"tag":74,"props":418,"children":421},{"href":419,"rel":420},"https:\u002F\u002Fanaconda.org\u002Fconda-forge\u002Fh5py",[77],[422],{"type":58,"value":423},"conda-forge build",{"type":58,"value":425},")",{"type":52,"tag":61,"props":427,"children":429},{"id":428},"api",[430],{"type":58,"value":431},"API",{"type":52,"tag":433,"props":434,"children":435},"table",{},[436,459],{"type":52,"tag":437,"props":438,"children":439},"thead",{},[440],{"type":52,"tag":441,"props":442,"children":443},"tr",{},[444,450,455],{"type":52,"tag":445,"props":446,"children":447},"th",{},[448],{"type":58,"value":449},"Function",{"type":52,"tag":445,"props":451,"children":452},{},[453],{"type":58,"value":454},"Signature",{"type":52,"tag":445,"props":456,"children":457},{},[458],{"type":58,"value":66},{"type":52,"tag":460,"props":461,"children":462},"tbody",{},[463,498,524],{"type":52,"tag":441,"props":464,"children":465},{},[466,476,485],{"type":52,"tag":467,"props":468,"children":469},"td",{},[470],{"type":52,"tag":80,"props":471,"children":473},{"className":472},[],[474],{"type":58,"value":475},"to_file",{"type":52,"tag":467,"props":477,"children":478},{},[479],{"type":52,"tag":80,"props":480,"children":482},{"className":481},[],[483],{"type":58,"value":484},"to_file(array, path, dataset_name)",{"type":52,"tag":467,"props":486,"children":487},{},[488,490,496],{"type":58,"value":489},"Write a cuPyNumeric array \u002F ",{"type":52,"tag":80,"props":491,"children":493},{"className":492},[],[494],{"type":58,"value":495},"LogicalArray",{"type":58,"value":497}," to one HDF5 file as a virtual dataset (VDS) — each rank writes its own tile.",{"type":52,"tag":441,"props":499,"children":500},{},[501,510,519],{"type":52,"tag":467,"props":502,"children":503},{},[504],{"type":52,"tag":80,"props":505,"children":507},{"className":506},[],[508],{"type":58,"value":509},"from_file",{"type":52,"tag":467,"props":511,"children":512},{},[513],{"type":52,"tag":80,"props":514,"children":516},{"className":515},[],[517],{"type":58,"value":518},"from_file(path, dataset_name) -> LogicalArray",{"type":52,"tag":467,"props":520,"children":521},{},[522],{"type":58,"value":523},"Read one HDF5 dataset into a distributed array.",{"type":52,"tag":441,"props":525,"children":526},{},[527,536,545],{"type":52,"tag":467,"props":528,"children":529},{},[530],{"type":52,"tag":80,"props":531,"children":533},{"className":532},[],[534],{"type":58,"value":535},"from_file_batched",{"type":52,"tag":467,"props":537,"children":538},{},[539],{"type":52,"tag":80,"props":540,"children":542},{"className":541},[],[543],{"type":58,"value":544},"from_file_batched(path, dataset_name, chunk_size) -> Iterator[(LogicalArray, offsets)]",{"type":52,"tag":467,"props":546,"children":547},{},[548],{"type":58,"value":549},"Read a dataset in chunks — chunks the file read, not the assembled array.",{"type":52,"tag":68,"props":551,"children":552},{},[553,555,560,562,568,570,576,578,584],{"type":58,"value":554},"Import all three from ",{"type":52,"tag":80,"props":556,"children":558},{"className":557},[],[559],{"type":58,"value":85},{"type":58,"value":561},". Always pass ",{"type":52,"tag":80,"props":563,"children":565},{"className":564},[],[566],{"type":58,"value":567},"dataset_name",{"type":58,"value":569}," as the full path to a single array inside the file (e.g. ",{"type":52,"tag":80,"props":571,"children":573},{"className":572},[],[574],{"type":58,"value":575},"\"\u002Fdata\"",{"type":58,"value":577}," or ",{"type":52,"tag":80,"props":579,"children":581},{"className":580},[],[582],{"type":58,"value":583},"\"\u002Fgroup\u002Fx\"",{"type":58,"value":585},"), never a group.",{"type":52,"tag":61,"props":587,"children":589},{"id":588},"examples",[590],{"type":58,"value":591},"Examples",{"type":52,"tag":593,"props":594,"children":596},"h3",{"id":595},"round-trip",[597],{"type":58,"value":598},"Round trip",{"type":52,"tag":332,"props":600,"children":603},{"className":601,"code":602,"language":19,"meta":337,"style":337},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import cupynumeric as cn\nfrom legate.core import get_legate_runtime\nfrom legate.io.hdf5 import from_file, to_file\n\na = cn.arange(64, dtype=cn.float32).reshape(8, 8)\n\n# Write: pass the cuPyNumeric ndarray straight in - no manual conversion.\nto_file(array=a, path=\"out.h5\", dataset_name=\"\u002Fdata\")\nget_legate_runtime().issue_execution_fence(block=True)   # needed before any external reader\n\n# Read: from_file returns a legate LogicalArray; cn.asarray bridges it back.\nb = cn.asarray(from_file(\"out.h5\", dataset_name=\"\u002Fdata\"))\nassert cn.array_equal(a, b)\n",[604],{"type":52,"tag":80,"props":605,"children":606},{"__ignoreMap":337},[607,615,624,633,643,652,660,669,678,687,695,704,713],{"type":52,"tag":343,"props":608,"children":609},{"class":345,"line":346},[610],{"type":52,"tag":343,"props":611,"children":612},{},[613],{"type":58,"value":614},"import cupynumeric as cn\n",{"type":52,"tag":343,"props":616,"children":618},{"class":345,"line":617},2,[619],{"type":52,"tag":343,"props":620,"children":621},{},[622],{"type":58,"value":623},"from legate.core import get_legate_runtime\n",{"type":52,"tag":343,"props":625,"children":627},{"class":345,"line":626},3,[628],{"type":52,"tag":343,"props":629,"children":630},{},[631],{"type":58,"value":632},"from legate.io.hdf5 import from_file, to_file\n",{"type":52,"tag":343,"props":634,"children":636},{"class":345,"line":635},4,[637],{"type":52,"tag":343,"props":638,"children":640},{"emptyLinePlaceholder":639},true,[641],{"type":58,"value":642},"\n",{"type":52,"tag":343,"props":644,"children":646},{"class":345,"line":645},5,[647],{"type":52,"tag":343,"props":648,"children":649},{},[650],{"type":58,"value":651},"a = cn.arange(64, dtype=cn.float32).reshape(8, 8)\n",{"type":52,"tag":343,"props":653,"children":655},{"class":345,"line":654},6,[656],{"type":52,"tag":343,"props":657,"children":658},{"emptyLinePlaceholder":639},[659],{"type":58,"value":642},{"type":52,"tag":343,"props":661,"children":663},{"class":345,"line":662},7,[664],{"type":52,"tag":343,"props":665,"children":666},{},[667],{"type":58,"value":668},"# Write: pass the cuPyNumeric ndarray straight in - no manual conversion.\n",{"type":52,"tag":343,"props":670,"children":672},{"class":345,"line":671},8,[673],{"type":52,"tag":343,"props":674,"children":675},{},[676],{"type":58,"value":677},"to_file(array=a, path=\"out.h5\", dataset_name=\"\u002Fdata\")\n",{"type":52,"tag":343,"props":679,"children":681},{"class":345,"line":680},9,[682],{"type":52,"tag":343,"props":683,"children":684},{},[685],{"type":58,"value":686},"get_legate_runtime().issue_execution_fence(block=True)   # needed before any external reader\n",{"type":52,"tag":343,"props":688,"children":690},{"class":345,"line":689},10,[691],{"type":52,"tag":343,"props":692,"children":693},{"emptyLinePlaceholder":639},[694],{"type":58,"value":642},{"type":52,"tag":343,"props":696,"children":698},{"class":345,"line":697},11,[699],{"type":52,"tag":343,"props":700,"children":701},{},[702],{"type":58,"value":703},"# Read: from_file returns a legate LogicalArray; cn.asarray bridges it back.\n",{"type":52,"tag":343,"props":705,"children":707},{"class":345,"line":706},12,[708],{"type":52,"tag":343,"props":709,"children":710},{},[711],{"type":58,"value":712},"b = cn.asarray(from_file(\"out.h5\", dataset_name=\"\u002Fdata\"))\n",{"type":52,"tag":343,"props":714,"children":716},{"class":345,"line":715},13,[717],{"type":52,"tag":343,"props":718,"children":719},{},[720],{"type":58,"value":721},"assert cn.array_equal(a, b)\n",{"type":52,"tag":68,"props":723,"children":724},{},[725,727,733],{"type":58,"value":726},"Run ",{"type":52,"tag":80,"props":728,"children":730},{"className":729},[],[731],{"type":58,"value":732},"assets\u002Fhdf5_roundtrip.py",{"type":58,"value":734}," to verify (optional — not needed to answer).",{"type":52,"tag":593,"props":736,"children":738},{"id":737},"read-a-large-file-in-chunks",[739],{"type":58,"value":740},"Read a large file in chunks",{"type":52,"tag":68,"props":742,"children":743},{},[744,745,750,752,757,759,765,767,773,775,780,782,788],{"type":58,"value":72},{"type":52,"tag":80,"props":746,"children":748},{"className":747},[],[749],{"type":58,"value":535},{"type":58,"value":751}," to read the source file in chunks instead of pulling it into host memory all at once. It yields one ",{"type":52,"tag":80,"props":753,"children":755},{"className":754},[],[756],{"type":58,"value":495},{"type":58,"value":758}," per chunk plus that chunk's offsets in the global shape. Expect clipped boundary chunks (an axis of length 5 with ",{"type":52,"tag":80,"props":760,"children":762},{"className":761},[],[763],{"type":58,"value":764},"chunk_size=2",{"type":58,"value":766}," yields 2, 2, 1), so place each chunk by its actual shape, not the requested ",{"type":52,"tag":80,"props":768,"children":770},{"className":769},[],[771],{"type":58,"value":772},"chunk_size",{"type":58,"value":774},". Note that this chunks the ",{"type":52,"tag":148,"props":776,"children":777},{},[778],{"type":58,"value":779},"file read",{"type":58,"value":781},", not the result — the assembled array (",{"type":52,"tag":80,"props":783,"children":785},{"className":784},[],[786],{"type":58,"value":787},"out",{"type":58,"value":789},") still has to fit in distributed memory:",{"type":52,"tag":332,"props":791,"children":793},{"className":601,"code":792,"language":19,"meta":337,"style":337},"import h5py\nimport cupynumeric as cn\nfrom legate.core import get_legate_runtime\nfrom legate.io.hdf5 import from_file_batched\n\nwith h5py.File(\"big.h5\", \"r\") as f:          # read shape\u002Fdtype without loading data\n    shape, dtype = f[\"data\"].shape, f[\"data\"].dtype\n\nout = cn.empty(shape, dtype=dtype)\nfor chunk, (r0, c0) in from_file_batched(\"big.h5\", \"data\", chunk_size=(4096, 4096)):\n    out[r0:r0 + chunk.shape[0], c0:c0 + chunk.shape[1]] = cn.asarray(chunk)\nget_legate_runtime().issue_execution_fence(block=True)\n",[794],{"type":52,"tag":80,"props":795,"children":796},{"__ignoreMap":337},[797,805,812,819,827,834,842,850,857,865,873,881],{"type":52,"tag":343,"props":798,"children":799},{"class":345,"line":346},[800],{"type":52,"tag":343,"props":801,"children":802},{},[803],{"type":58,"value":804},"import h5py\n",{"type":52,"tag":343,"props":806,"children":807},{"class":345,"line":617},[808],{"type":52,"tag":343,"props":809,"children":810},{},[811],{"type":58,"value":614},{"type":52,"tag":343,"props":813,"children":814},{"class":345,"line":626},[815],{"type":52,"tag":343,"props":816,"children":817},{},[818],{"type":58,"value":623},{"type":52,"tag":343,"props":820,"children":821},{"class":345,"line":635},[822],{"type":52,"tag":343,"props":823,"children":824},{},[825],{"type":58,"value":826},"from legate.io.hdf5 import from_file_batched\n",{"type":52,"tag":343,"props":828,"children":829},{"class":345,"line":645},[830],{"type":52,"tag":343,"props":831,"children":832},{"emptyLinePlaceholder":639},[833],{"type":58,"value":642},{"type":52,"tag":343,"props":835,"children":836},{"class":345,"line":654},[837],{"type":52,"tag":343,"props":838,"children":839},{},[840],{"type":58,"value":841},"with h5py.File(\"big.h5\", \"r\") as f:          # read shape\u002Fdtype without loading data\n",{"type":52,"tag":343,"props":843,"children":844},{"class":345,"line":662},[845],{"type":52,"tag":343,"props":846,"children":847},{},[848],{"type":58,"value":849},"    shape, dtype = f[\"data\"].shape, f[\"data\"].dtype\n",{"type":52,"tag":343,"props":851,"children":852},{"class":345,"line":671},[853],{"type":52,"tag":343,"props":854,"children":855},{"emptyLinePlaceholder":639},[856],{"type":58,"value":642},{"type":52,"tag":343,"props":858,"children":859},{"class":345,"line":680},[860],{"type":52,"tag":343,"props":861,"children":862},{},[863],{"type":58,"value":864},"out = cn.empty(shape, dtype=dtype)\n",{"type":52,"tag":343,"props":866,"children":867},{"class":345,"line":689},[868],{"type":52,"tag":343,"props":869,"children":870},{},[871],{"type":58,"value":872},"for chunk, (r0, c0) in from_file_batched(\"big.h5\", \"data\", chunk_size=(4096, 4096)):\n",{"type":52,"tag":343,"props":874,"children":875},{"class":345,"line":697},[876],{"type":52,"tag":343,"props":877,"children":878},{},[879],{"type":58,"value":880},"    out[r0:r0 + chunk.shape[0], c0:c0 + chunk.shape[1]] = cn.asarray(chunk)\n",{"type":52,"tag":343,"props":882,"children":883},{"class":345,"line":706},[884],{"type":52,"tag":343,"props":885,"children":886},{},[887],{"type":58,"value":888},"get_legate_runtime().issue_execution_fence(block=True)\n",{"type":52,"tag":68,"props":890,"children":891},{},[892,894,899,901,906,908,914,916,922],{"type":58,"value":893},"Keep every ",{"type":52,"tag":80,"props":895,"children":897},{"className":896},[],[898],{"type":58,"value":772},{"type":58,"value":900}," entry positive and its length equal to the dataset's rank, or ",{"type":52,"tag":80,"props":902,"children":904},{"className":903},[],[905],{"type":58,"value":535},{"type":58,"value":907}," raises ",{"type":52,"tag":80,"props":909,"children":911},{"className":910},[],[912],{"type":58,"value":913},"ValueError",{"type":58,"value":915},". Run ",{"type":52,"tag":80,"props":917,"children":919},{"className":918},[],[920],{"type":58,"value":921},"assets\u002Fhdf5_batched_read.py",{"type":58,"value":923}," to verify (optional).",{"type":52,"tag":61,"props":925,"children":927},{"id":926},"instructions",[928],{"type":58,"value":929},"Instructions",{"type":52,"tag":199,"props":931,"children":932},{},[933,979,1024,1056,1098],{"type":52,"tag":203,"props":934,"children":935},{},[936,946,948,954,956,961,963,969,971,977],{"type":52,"tag":125,"props":937,"children":938},{},[939,941],{"type":58,"value":940},"Pass the cuPyNumeric ndarray directly to ",{"type":52,"tag":80,"props":942,"children":944},{"className":943},[],[945],{"type":58,"value":475},{"type":58,"value":947}," - it implements ",{"type":52,"tag":80,"props":949,"children":951},{"className":950},[],[952],{"type":58,"value":953},"__legate_data_interface__",{"type":58,"value":955},", which ",{"type":52,"tag":80,"props":957,"children":959},{"className":958},[],[960],{"type":58,"value":475},{"type":58,"value":962}," accepts as ",{"type":52,"tag":80,"props":964,"children":966},{"className":965},[],[967],{"type":58,"value":968},"LogicalArrayLike",{"type":58,"value":970},". Skip any ",{"type":52,"tag":80,"props":972,"children":974},{"className":973},[],[975],{"type":58,"value":976},"np.array(...)",{"type":58,"value":978}," round-trip.",{"type":52,"tag":203,"props":980,"children":981},{},[982,993,995,1000,1002,1007,1009,1014,1016,1022],{"type":52,"tag":125,"props":983,"children":984},{},[985,987,992],{"type":58,"value":986},"Bridge results back with ",{"type":52,"tag":80,"props":988,"children":990},{"className":989},[],[991],{"type":58,"value":271},{"type":58,"value":313},{"type":58,"value":994}," ",{"type":52,"tag":80,"props":996,"children":998},{"className":997},[],[999],{"type":58,"value":509},{"type":58,"value":1001}," and each ",{"type":52,"tag":80,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":58,"value":535},{"type":58,"value":1008}," chunk return a Legate ",{"type":52,"tag":80,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":58,"value":495},{"type":58,"value":1015},"; wrap it with ",{"type":52,"tag":80,"props":1017,"children":1019},{"className":1018},[],[1020],{"type":58,"value":1021},"cn.asarray(la)",{"type":58,"value":1023}," to get a cuPyNumeric ndarray (zero-copy, no host bounce).",{"type":52,"tag":203,"props":1025,"children":1026},{},[1027,1032,1034,1039,1041,1047,1049,1054],{"type":52,"tag":125,"props":1028,"children":1029},{},[1030],{"type":58,"value":1031},"Fence before any external reader.",{"type":58,"value":1033}," Legate I\u002FO is asynchronous: ",{"type":52,"tag":80,"props":1035,"children":1037},{"className":1036},[],[1038],{"type":58,"value":475},{"type":58,"value":1040}," only queues the write. Insert ",{"type":52,"tag":80,"props":1042,"children":1044},{"className":1043},[],[1045],{"type":58,"value":1046},"get_legate_runtime().issue_execution_fence(block=True)",{"type":58,"value":1048}," before h5py, a subprocess, or another tool opens the file. Skip the fence for a ",{"type":52,"tag":80,"props":1050,"children":1052},{"className":1051},[],[1053],{"type":58,"value":509},{"type":58,"value":1055},"\nissued later in the same Legate program — the runtime preserves that ordering.",{"type":52,"tag":203,"props":1057,"children":1058},{},[1059,1064,1066,1072,1074,1080,1082,1088,1090,1096],{"type":52,"tag":125,"props":1060,"children":1061},{},[1062],{"type":58,"value":1063},"Run from outside the cuPyNumeric source tree",{"type":58,"value":1065}," (e.g. ",{"type":52,"tag":80,"props":1067,"children":1069},{"className":1068},[],[1070],{"type":58,"value":1071},"cd \u002Ftmp",{"type":58,"value":1073},"). Python puts the cwd first on ",{"type":52,"tag":80,"props":1075,"children":1077},{"className":1076},[],[1078],{"type":58,"value":1079},"sys.path",{"type":58,"value":1081},", so an in-tree ",{"type":52,"tag":80,"props":1083,"children":1085},{"className":1084},[],[1086],{"type":58,"value":1087},"cupynumeric\u002F",{"type":58,"value":1089}," directory shadows the installed package (",{"type":52,"tag":80,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":58,"value":1095},"ModuleNotFoundError: cupynumeric.install_info",{"type":58,"value":1097},").",{"type":52,"tag":203,"props":1099,"children":1100},{},[1101,1113,1115,1120,1121,1126,1128,1133,1135,1141,1143,1148,1150,1155],{"type":52,"tag":125,"props":1102,"children":1103},{},[1104,1106,1112],{"type":58,"value":1105},"Give every rank the same ",{"type":52,"tag":80,"props":1107,"children":1109},{"className":1108},[],[1110],{"type":58,"value":1111},"path",{"type":58,"value":313},{"type":58,"value":1114}," The program runs on every rank (SPMD), so pass ",{"type":52,"tag":80,"props":1116,"children":1118},{"className":1117},[],[1119],{"type":58,"value":475},{"type":58,"value":112},{"type":52,"tag":80,"props":1122,"children":1124},{"className":1123},[],[1125],{"type":58,"value":509},{"type":58,"value":1127}," an identical ",{"type":52,"tag":80,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":58,"value":1111},{"type":58,"value":1134}," on each — a per-rank ",{"type":52,"tag":80,"props":1136,"children":1138},{"className":1137},[],[1139],{"type":58,"value":1140},"tempfile.mkstemp()",{"type":58,"value":1142}," name breaks the collective I\u002FO. When the program creates the file itself, write it with the collective ",{"type":52,"tag":80,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":58,"value":475},{"type":58,"value":1149},", not a per-rank ",{"type":52,"tag":80,"props":1151,"children":1153},{"className":1152},[],[1154],{"type":58,"value":42},{"type":58,"value":1156}," write.",{"type":52,"tag":61,"props":1158,"children":1160},{"id":1159},"to_file-behavior-to-plan-around",[1161,1166],{"type":52,"tag":80,"props":1162,"children":1164},{"className":1163},[],[1165],{"type":58,"value":475},{"type":58,"value":1167}," behavior to plan around",{"type":52,"tag":199,"props":1169,"children":1170},{},[1171,1183,1209,1227],{"type":52,"tag":203,"props":1172,"children":1173},{},[1174,1176,1181],{"type":58,"value":1175},"Expect an HDF5 ",{"type":52,"tag":125,"props":1177,"children":1178},{},[1179],{"type":58,"value":1180},"virtual dataset (VDS)",{"type":58,"value":1182},": each rank writes its own tile and the file presents them as one logical dataset.",{"type":52,"tag":203,"props":1184,"children":1185},{},[1186,1188,1193,1195,1200,1202,1207],{"type":58,"value":1187},"Treat ",{"type":52,"tag":80,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":58,"value":475},{"type":58,"value":1194}," as ",{"type":52,"tag":125,"props":1196,"children":1197},{},[1198],{"type":58,"value":1199},"destructive",{"type":58,"value":1201}," — it overwrites ",{"type":52,"tag":80,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":58,"value":1111},{"type":58,"value":1208}," if it already exists, so guard any file you must not clobber.",{"type":52,"tag":203,"props":1210,"children":1211},{},[1212,1214,1219,1220,1225],{"type":58,"value":1213},"Let ",{"type":52,"tag":80,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":58,"value":475},{"type":58,"value":994},{"type":52,"tag":125,"props":1221,"children":1222},{},[1223],{"type":58,"value":1224},"create missing parent directories",{"type":58,"value":1226},"; do not pre-create them.",{"type":52,"tag":203,"props":1228,"children":1229},{},[1230,1232,1237,1239,1245,1247,1252,1254,1259,1261,1266,1267,1272,1274,1279,1281,1287,1289,1294],{"type":58,"value":1231},"Give ",{"type":52,"tag":80,"props":1233,"children":1235},{"className":1234},[],[1236],{"type":58,"value":1111},{"type":58,"value":1238}," a file name (",{"type":52,"tag":80,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":58,"value":1244},"\u002Fpath\u002Fto\u002Ffile.h5",{"type":58,"value":1246},"), never a directory — a directory raises ",{"type":52,"tag":80,"props":1248,"children":1250},{"className":1249},[],[1251],{"type":58,"value":913},{"type":58,"value":1253},". Pass a ",{"type":52,"tag":125,"props":1255,"children":1256},{},[1257],{"type":58,"value":1258},"bound",{"type":58,"value":1260}," array (one with a known shape); ",{"type":52,"tag":80,"props":1262,"children":1264},{"className":1263},[],[1265],{"type":58,"value":475},{"type":58,"value":907},{"type":52,"tag":80,"props":1268,"children":1270},{"className":1269},[],[1271],{"type":58,"value":913},{"type":58,"value":1273}," on an ",{"type":52,"tag":148,"props":1275,"children":1276},{},[1277],{"type":58,"value":1278},"unbound",{"type":58,"value":1280}," array — a Legate array created without a shape (e.g. ",{"type":52,"tag":80,"props":1282,"children":1284},{"className":1283},[],[1285],{"type":58,"value":1286},"create_array(dtype, ndim=n)",{"type":58,"value":1288},") whose extent a producing task fills in later. cuPyNumeric ndarrays are always bound — even lazy\u002Fdeferred ones — so this only affects raw ",{"type":52,"tag":80,"props":1290,"children":1292},{"className":1291},[],[1293],{"type":58,"value":495},{"type":58,"value":1295},"s.",{"type":52,"tag":61,"props":1297,"children":1299},{"id":1298},"gpudirect-storage-gds",[1300],{"type":58,"value":1301},"GPUDirect Storage (GDS)",{"type":52,"tag":68,"props":1303,"children":1304},{},[1305,1318],{"type":52,"tag":125,"props":1306,"children":1307},{},[1308,1310,1316],{"type":58,"value":1309},"Always set ",{"type":52,"tag":80,"props":1311,"children":1313},{"className":1312},[],[1314],{"type":58,"value":1315},"LEGATE_IO_USE_VFD_GDS=1",{"type":58,"value":1317}," for runs that read HDF5 into GPU memory",{"type":58,"value":1319}," — whether or not the cluster has GPUDirect-capable storage:",{"type":52,"tag":332,"props":1321,"children":1323},{"className":334,"code":1322,"language":336,"meta":337,"style":337},"export LEGATE_IO_USE_VFD_GDS=1          # set before launching\n# or, with the legate driver:\nlegate --io-use-vfd-gds my_script.py\n",[1324],{"type":52,"tag":80,"props":1325,"children":1326},{"__ignoreMap":337},[1327,1359,1367],{"type":52,"tag":343,"props":1328,"children":1329},{"class":345,"line":346},[1330,1336,1342,1348,1354],{"type":52,"tag":343,"props":1331,"children":1333},{"style":1332},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1334],{"type":58,"value":1335},"export",{"type":52,"tag":343,"props":1337,"children":1339},{"style":1338},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1340],{"type":58,"value":1341}," LEGATE_IO_USE_VFD_GDS",{"type":52,"tag":343,"props":1343,"children":1345},{"style":1344},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1346],{"type":58,"value":1347},"=",{"type":52,"tag":343,"props":1349,"children":1351},{"style":1350},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1352],{"type":58,"value":1353},"1",{"type":52,"tag":343,"props":1355,"children":1356},{"style":377},[1357],{"type":58,"value":1358},"          # set before launching\n",{"type":52,"tag":343,"props":1360,"children":1361},{"class":345,"line":617},[1362],{"type":52,"tag":343,"props":1363,"children":1364},{"style":377},[1365],{"type":58,"value":1366},"# or, with the legate driver:\n",{"type":52,"tag":343,"props":1368,"children":1369},{"class":345,"line":626},[1370,1374,1379],{"type":52,"tag":343,"props":1371,"children":1372},{"style":350},[1373],{"type":58,"value":40},{"type":52,"tag":343,"props":1375,"children":1376},{"style":356},[1377],{"type":58,"value":1378}," --io-use-vfd-gds",{"type":52,"tag":343,"props":1380,"children":1381},{"style":356},[1382],{"type":58,"value":1383}," my_script.py\n",{"type":52,"tag":199,"props":1385,"children":1386},{},[1387,1397,1407,1440],{"type":52,"tag":203,"props":1388,"children":1389},{},[1390,1395],{"type":52,"tag":125,"props":1391,"children":1392},{},[1393],{"type":58,"value":1394},"Read into the GPU through the GDS VFD, not the default path.",{"type":58,"value":1396}," The default (POSIX) VFD stages each GPU read through zero-copy memory (ZCMEM), of which Legate reserves only 128 MB — so a GPU read of an array larger than ~128 MB aborts. The GDS VFD removes that staging buffer.",{"type":52,"tag":203,"props":1398,"children":1399},{},[1400,1405],{"type":52,"tag":125,"props":1401,"children":1402},{},[1403],{"type":58,"value":1404},"Leave it unset when reading into host (CPU) memory",{"type":58,"value":1406}," — the VFD GDS plugin is unnecessary there and only adds overhead.",{"type":52,"tag":203,"props":1408,"children":1409},{},[1410,1423,1425,1431,1433,1438],{"type":52,"tag":125,"props":1411,"children":1412},{},[1413,1415,1421],{"type":58,"value":1414},"Keep ",{"type":52,"tag":80,"props":1416,"children":1418},{"className":1417},[],[1419],{"type":58,"value":1420},"=1",{"type":58,"value":1422}," even without GPUDirect-capable storage",{"type":58,"value":1424}," — cuFile falls back to compatibility mode automatically (set ",{"type":52,"tag":80,"props":1426,"children":1428},{"className":1427},[],[1429],{"type":58,"value":1430},"export CUFILE_ALLOW_COMPAT_MODE=true",{"type":58,"value":1432}," if it is not already on), and ",{"type":52,"tag":80,"props":1434,"children":1436},{"className":1435},[],[1437],{"type":58,"value":1420},{"type":58,"value":1439}," still avoids the ZCMEM abort.",{"type":52,"tag":203,"props":1441,"children":1442},{},[1443,1448,1450,1457,1459,1466,1468,1473,1475,1481],{"type":52,"tag":125,"props":1444,"children":1445},{},[1446],{"type":58,"value":1447},"Attribute it correctly:",{"type":58,"value":1449}," the GDS VFD is the ",{"type":52,"tag":74,"props":1451,"children":1454},{"href":1452,"rel":1453},"https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fvfd-gds",[77],[1455],{"type":58,"value":1456},"nv-legate\u002Fvfd-gds",{"type":58,"value":1458}," plugin over NVIDIA ",{"type":52,"tag":74,"props":1460,"children":1463},{"href":1461,"rel":1462},"https:\u002F\u002Fdeveloper.nvidia.com\u002Fgpudirect-storage",[77],[1464],{"type":58,"value":1465},"cuFile",{"type":58,"value":1467},", ",{"type":52,"tag":125,"props":1469,"children":1470},{},[1471],{"type":58,"value":1472},"not",{"type":58,"value":1474}," KvikIO (KvikIO backs Legate's Zarr\u002Ftile I\u002FO, not HDF5). Confirm it engaged by grepping the run log for ",{"type":52,"tag":80,"props":1476,"children":1478},{"className":1477},[],[1479],{"type":58,"value":1480},"H5FD__gds_open: Successfully opened file w\u002FGDS VFD",{"type":58,"value":313},{"type":52,"tag":61,"props":1483,"children":1485},{"id":1484},"troubleshooting",[1486],{"type":58,"value":1487},"Troubleshooting",{"type":52,"tag":433,"props":1489,"children":1490},{},[1491,1507],{"type":52,"tag":437,"props":1492,"children":1493},{},[1494],{"type":52,"tag":441,"props":1495,"children":1496},{},[1497,1502],{"type":52,"tag":445,"props":1498,"children":1499},{},[1500],{"type":58,"value":1501},"Symptom",{"type":52,"tag":445,"props":1503,"children":1504},{},[1505],{"type":58,"value":1506},"Cause and fix",{"type":52,"tag":460,"props":1508,"children":1509},{},[1510,1536,1561,1596,1620,1640],{"type":52,"tag":441,"props":1511,"children":1512},{},[1513,1524],{"type":52,"tag":467,"props":1514,"children":1515},{},[1516,1522],{"type":52,"tag":80,"props":1517,"children":1519},{"className":1518},[],[1520],{"type":58,"value":1521},"ModuleNotFoundError: No module named 'h5py'",{"type":58,"value":1523}," on import",{"type":52,"tag":467,"props":1525,"children":1526},{},[1527,1529,1535],{"type":58,"value":1528},"h5py is missing — ",{"type":52,"tag":80,"props":1530,"children":1532},{"className":1531},[],[1533],{"type":58,"value":1534},"conda install -c conda-forge h5py",{"type":58,"value":313},{"type":52,"tag":441,"props":1537,"children":1538},{},[1539,1549],{"type":52,"tag":467,"props":1540,"children":1541},{},[1542,1544],{"type":58,"value":1543},"File looks empty\u002Ftruncated to h5py right after ",{"type":52,"tag":80,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":58,"value":475},{"type":52,"tag":467,"props":1550,"children":1551},{},[1552,1554,1559],{"type":58,"value":1553},"The async write hasn't landed — add ",{"type":52,"tag":80,"props":1555,"children":1557},{"className":1556},[],[1558],{"type":58,"value":1046},{"type":58,"value":1560}," before the external read.",{"type":52,"tag":441,"props":1562,"children":1563},{},[1564,1579],{"type":52,"tag":467,"props":1565,"children":1566},{},[1567,1572,1574],{"type":52,"tag":80,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":58,"value":913},{"type":58,"value":1573}," from ",{"type":52,"tag":80,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":58,"value":475},{"type":52,"tag":467,"props":1580,"children":1581},{},[1582,1587,1589,1595],{"type":52,"tag":80,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":58,"value":1111},{"type":58,"value":1588}," is a directory — pass a file path such as ",{"type":52,"tag":80,"props":1590,"children":1592},{"className":1591},[],[1593],{"type":58,"value":1594},"results\u002Fdata.h5",{"type":58,"value":313},{"type":52,"tag":441,"props":1597,"children":1598},{},[1599,1608],{"type":52,"tag":467,"props":1600,"children":1601},{},[1602],{"type":52,"tag":80,"props":1603,"children":1605},{"className":1604},[],[1606],{"type":58,"value":1607},"ModuleNotFoundError: No module named 'cupynumeric.install_info'",{"type":52,"tag":467,"props":1609,"children":1610},{},[1611,1613,1618],{"type":58,"value":1612},"Running inside the source tree — ",{"type":52,"tag":80,"props":1614,"children":1616},{"className":1615},[],[1617],{"type":58,"value":1071},{"type":58,"value":1619}," (any directory outside the repo).",{"type":52,"tag":441,"props":1621,"children":1622},{},[1623,1628],{"type":52,"tag":467,"props":1624,"children":1625},{},[1626],{"type":58,"value":1627},"Abort\u002Fcrash reading a GPU array ≳128 MB",{"type":52,"tag":467,"props":1629,"children":1630},{},[1631,1633,1638],{"type":58,"value":1632},"Default 128 MB ZCMEM staging buffer — set ",{"type":52,"tag":80,"props":1634,"children":1636},{"className":1635},[],[1637],{"type":58,"value":1315},{"type":58,"value":1639}," for GPU reads.",{"type":52,"tag":441,"props":1641,"children":1642},{},[1643,1659],{"type":52,"tag":467,"props":1644,"children":1645},{},[1646,1651,1653],{"type":52,"tag":80,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":58,"value":509},{"type":58,"value":1652}," returned ",{"type":52,"tag":80,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":58,"value":1658},"LogicalArray(...)",{"type":52,"tag":467,"props":1660,"children":1661},{},[1662,1664,1669],{"type":58,"value":1663},"Expected — wrap it with ",{"type":52,"tag":80,"props":1665,"children":1667},{"className":1666},[],[1668],{"type":58,"value":271},{"type":58,"value":313},{"type":52,"tag":61,"props":1671,"children":1673},{"id":1672},"limitations-version-notes",[1674],{"type":58,"value":1675},"Limitations & version notes",{"type":52,"tag":199,"props":1677,"children":1678},{},[1679,1711,1721,1738],{"type":52,"tag":203,"props":1680,"children":1681},{},[1682,1692,1694,1700,1702,1709],{"type":52,"tag":125,"props":1683,"children":1684},{},[1685,1687],{"type":58,"value":1686},"Import from ",{"type":52,"tag":80,"props":1688,"children":1690},{"className":1689},[],[1691],{"type":58,"value":85},{"type":58,"value":1693}," (Legate 26.01+); rewrite any ",{"type":52,"tag":80,"props":1695,"children":1697},{"className":1696},[],[1698],{"type":58,"value":1699},"legate.core.io.hdf5",{"type":58,"value":1701}," import left over from the 25.03 line (e.g. the ",{"type":52,"tag":74,"props":1703,"children":1706},{"href":1704,"rel":1705},"https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fnvidia-cupynumeric-25-03-now-fully-open-source-with-pip-and-hdf5-support\u002F",[77],[1707],{"type":58,"value":1708},"25.03 launch blog",{"type":58,"value":1710}," still shows the old path).",{"type":52,"tag":203,"props":1712,"children":1713},{},[1714,1719],{"type":52,"tag":125,"props":1715,"children":1716},{},[1717],{"type":58,"value":1718},"Install h5py explicitly",{"type":58,"value":1720}," — it ships in no default cuPyNumeric env.",{"type":52,"tag":203,"props":1722,"children":1723},{},[1724,1736],{"type":52,"tag":125,"props":1725,"children":1726},{},[1727,1729,1734],{"type":58,"value":1728},"Point ",{"type":52,"tag":80,"props":1730,"children":1732},{"className":1731},[],[1733],{"type":58,"value":567},{"type":58,"value":1735}," at a single array, never a group",{"type":58,"value":1737},"; traverse groups with h5py first to discover dataset paths.",{"type":52,"tag":203,"props":1739,"children":1740},{},[1741,1751,1753,1759],{"type":52,"tag":125,"props":1742,"children":1743},{},[1744,1746],{"type":58,"value":1745},"On GPU, always read with ",{"type":52,"tag":80,"props":1747,"children":1749},{"className":1748},[],[1750],{"type":58,"value":1315},{"type":58,"value":1752}," (see ",{"type":52,"tag":74,"props":1754,"children":1756},{"href":1755},"#gpudirect-storage-gds",[1757],{"type":58,"value":1758},"GPUDirect Storage",{"type":58,"value":1760},") — the default path aborts on GPU arrays larger than the 128 MB ZCMEM buffer. Leave it unset for CPU reads.",{"type":52,"tag":61,"props":1762,"children":1764},{"id":1763},"verify",[1765],{"type":58,"value":1766},"Verify",{"type":52,"tag":332,"props":1768,"children":1770},{"className":334,"code":1769,"language":336,"meta":337,"style":337},"cd \u002Ftmp                                  # outside the cupynumeric source tree\nconda install -c conda-forge h5py        # one-time, if not already present\nLEGATE_CONFIG=\"--cpus 4\" LEGATE_AUTO_CONFIG=0 python \u003Cskill>\u002Fassets\u002Fhdf5_roundtrip.py\nLEGATE_CONFIG=\"--cpus 4\" LEGATE_AUTO_CONFIG=0 python \u003Cskill>\u002Fassets\u002Fhdf5_batched_read.py\n",[1771],{"type":52,"tag":80,"props":1772,"children":1773},{"__ignoreMap":337},[1774,1793,1821,1891],{"type":52,"tag":343,"props":1775,"children":1776},{"class":345,"line":346},[1777,1783,1788],{"type":52,"tag":343,"props":1778,"children":1780},{"style":1779},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1781],{"type":58,"value":1782},"cd",{"type":52,"tag":343,"props":1784,"children":1785},{"style":356},[1786],{"type":58,"value":1787}," \u002Ftmp",{"type":52,"tag":343,"props":1789,"children":1790},{"style":377},[1791],{"type":58,"value":1792},"                                  # outside the cupynumeric source tree\n",{"type":52,"tag":343,"props":1794,"children":1795},{"class":345,"line":617},[1796,1800,1804,1808,1812,1816],{"type":52,"tag":343,"props":1797,"children":1798},{"style":350},[1799],{"type":58,"value":353},{"type":52,"tag":343,"props":1801,"children":1802},{"style":356},[1803],{"type":58,"value":359},{"type":52,"tag":343,"props":1805,"children":1806},{"style":356},[1807],{"type":58,"value":364},{"type":52,"tag":343,"props":1809,"children":1810},{"style":356},[1811],{"type":58,"value":369},{"type":52,"tag":343,"props":1813,"children":1814},{"style":356},[1815],{"type":58,"value":374},{"type":52,"tag":343,"props":1817,"children":1818},{"style":377},[1819],{"type":58,"value":1820},"        # one-time, if not already present\n",{"type":52,"tag":343,"props":1822,"children":1823},{"class":345,"line":626},[1824,1829,1833,1838,1843,1847,1852,1856,1861,1866,1871,1876,1881,1886],{"type":52,"tag":343,"props":1825,"children":1826},{"style":1338},[1827],{"type":58,"value":1828},"LEGATE_CONFIG",{"type":52,"tag":343,"props":1830,"children":1831},{"style":1344},[1832],{"type":58,"value":1347},{"type":52,"tag":343,"props":1834,"children":1835},{"style":1344},[1836],{"type":58,"value":1837},"\"",{"type":52,"tag":343,"props":1839,"children":1840},{"style":356},[1841],{"type":58,"value":1842},"--cpus 4",{"type":52,"tag":343,"props":1844,"children":1845},{"style":1344},[1846],{"type":58,"value":1837},{"type":52,"tag":343,"props":1848,"children":1849},{"style":1338},[1850],{"type":58,"value":1851}," LEGATE_AUTO_CONFIG",{"type":52,"tag":343,"props":1853,"children":1854},{"style":1344},[1855],{"type":58,"value":1347},{"type":52,"tag":343,"props":1857,"children":1858},{"style":356},[1859],{"type":58,"value":1860},"0",{"type":52,"tag":343,"props":1862,"children":1863},{"style":350},[1864],{"type":58,"value":1865}," python",{"type":52,"tag":343,"props":1867,"children":1868},{"style":1344},[1869],{"type":58,"value":1870}," \u003C",{"type":52,"tag":343,"props":1872,"children":1873},{"style":356},[1874],{"type":58,"value":1875},"skil",{"type":52,"tag":343,"props":1877,"children":1878},{"style":1338},[1879],{"type":58,"value":1880},"l",{"type":52,"tag":343,"props":1882,"children":1883},{"style":1344},[1884],{"type":58,"value":1885},">",{"type":52,"tag":343,"props":1887,"children":1888},{"style":356},[1889],{"type":58,"value":1890},"\u002Fassets\u002Fhdf5_roundtrip.py\n",{"type":52,"tag":343,"props":1892,"children":1893},{"class":345,"line":635},[1894,1898,1902,1906,1910,1914,1918,1922,1926,1930,1934,1938,1942,1946],{"type":52,"tag":343,"props":1895,"children":1896},{"style":1338},[1897],{"type":58,"value":1828},{"type":52,"tag":343,"props":1899,"children":1900},{"style":1344},[1901],{"type":58,"value":1347},{"type":52,"tag":343,"props":1903,"children":1904},{"style":1344},[1905],{"type":58,"value":1837},{"type":52,"tag":343,"props":1907,"children":1908},{"style":356},[1909],{"type":58,"value":1842},{"type":52,"tag":343,"props":1911,"children":1912},{"style":1344},[1913],{"type":58,"value":1837},{"type":52,"tag":343,"props":1915,"children":1916},{"style":1338},[1917],{"type":58,"value":1851},{"type":52,"tag":343,"props":1919,"children":1920},{"style":1344},[1921],{"type":58,"value":1347},{"type":52,"tag":343,"props":1923,"children":1924},{"style":356},[1925],{"type":58,"value":1860},{"type":52,"tag":343,"props":1927,"children":1928},{"style":350},[1929],{"type":58,"value":1865},{"type":52,"tag":343,"props":1931,"children":1932},{"style":1344},[1933],{"type":58,"value":1870},{"type":52,"tag":343,"props":1935,"children":1936},{"style":356},[1937],{"type":58,"value":1875},{"type":52,"tag":343,"props":1939,"children":1940},{"style":1338},[1941],{"type":58,"value":1880},{"type":52,"tag":343,"props":1943,"children":1944},{"style":1344},[1945],{"type":58,"value":1885},{"type":52,"tag":343,"props":1947,"children":1948},{"style":356},[1949],{"type":58,"value":1950},"\u002Fassets\u002Fhdf5_batched_read.py\n",{"type":52,"tag":68,"props":1952,"children":1953},{},[1954,1955,1961,1963,1969,1971,1977,1979,1984],{"type":58,"value":385},{"type":52,"tag":80,"props":1956,"children":1958},{"className":1957},[],[1959],{"type":58,"value":1960},"HDF5 ROUND TRIP OK",{"type":58,"value":1962}," and ",{"type":52,"tag":80,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":58,"value":1968},"HDF5 BATCHED READ OK",{"type":58,"value":1970},". Add ",{"type":52,"tag":80,"props":1972,"children":1974},{"className":1973},[],[1975],{"type":58,"value":1976},"--gpus 1",{"type":58,"value":1978}," (and ",{"type":52,"tag":80,"props":1980,"children":1982},{"className":1981},[],[1983],{"type":58,"value":1315},{"type":58,"value":1985},") to exercise the GPU \u002F GDS path.",{"type":52,"tag":1987,"props":1988,"children":1989},"style",{},[1990],{"type":58,"value":1991},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1993,"total":2095},[1994,2009,2023,2037,2049,2066,2081],{"slug":1995,"name":1995,"fn":1996,"description":1997,"org":1998,"tags":1999,"stars":20,"repoUrl":21,"updatedAt":2008},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2000,2003,2004,2005],{"name":2001,"slug":2002,"type":15},"Data Analysis","data-analysis",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":2006,"slug":2007,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":2010,"name":2010,"fn":2011,"description":2012,"org":2013,"tags":2014,"stars":20,"repoUrl":21,"updatedAt":2022},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2015,2018,2021],{"name":2016,"slug":2017,"type":15},"Deployment","deployment",{"name":2019,"slug":2020,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2024,"name":2024,"fn":2025,"description":2026,"org":2027,"tags":2028,"stars":20,"repoUrl":21,"updatedAt":2036},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2029,2032,2033],{"name":2030,"slug":2031,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2034,"slug":2035,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2038,"name":2038,"fn":2039,"description":2040,"org":2041,"tags":2042,"stars":20,"repoUrl":21,"updatedAt":2048},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2043,2044,2045],{"name":2001,"slug":2002,"type":15},{"name":9,"slug":8,"type":15},{"name":2046,"slug":2047,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2050,"name":2050,"fn":2051,"description":2052,"org":2053,"tags":2054,"stars":20,"repoUrl":21,"updatedAt":2065},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2055,2058,2061,2062],{"name":2056,"slug":2057,"type":15},"Automation","automation",{"name":2059,"slug":2060,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2063,"slug":2064,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2067,"name":2067,"fn":2068,"description":2069,"org":2070,"tags":2071,"stars":20,"repoUrl":21,"updatedAt":2080},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2072,2073,2076,2077],{"name":2016,"slug":2017,"type":15},{"name":2074,"slug":2075,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2078,"slug":2079,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2082,"name":2082,"fn":2083,"description":2084,"org":2085,"tags":2086,"stars":20,"repoUrl":21,"updatedAt":2094},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2087,2088,2091],{"name":9,"slug":8,"type":15},{"name":2089,"slug":2090,"type":15},"Quantum Computing","quantum-computing",{"name":2092,"slug":2093,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2097,"total":2246},[2098,2116,2130,2141,2153,2167,2180,2194,2205,2214,2228,2237],{"slug":2099,"name":2099,"fn":2100,"description":2101,"org":2102,"tags":2103,"stars":2113,"repoUrl":2114,"updatedAt":2115},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2104,2107,2110],{"name":2105,"slug":2106,"type":15},"Documentation","documentation",{"name":2108,"slug":2109,"type":15},"MCP","mcp",{"name":2111,"slug":2112,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2117,"name":2117,"fn":2118,"description":2119,"org":2120,"tags":2121,"stars":2127,"repoUrl":2128,"updatedAt":2129},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2122,2125,2126],{"name":2123,"slug":2124,"type":15},"Containers","containers",{"name":2016,"slug":2017,"type":15},{"name":18,"slug":19,"type":15},17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2131,"name":2131,"fn":2132,"description":2133,"org":2134,"tags":2135,"stars":2127,"repoUrl":2128,"updatedAt":2140},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2136,2139],{"name":2137,"slug":2138,"type":15},"CI\u002FCD","ci-cd",{"name":2016,"slug":2017,"type":15},"2026-07-14T05:25:59.97109",{"slug":2142,"name":2142,"fn":2143,"description":2144,"org":2145,"tags":2146,"stars":2127,"repoUrl":2128,"updatedAt":2152},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2147,2148,2149],{"name":2137,"slug":2138,"type":15},{"name":2016,"slug":2017,"type":15},{"name":2150,"slug":2151,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2154,"name":2154,"fn":2155,"description":2156,"org":2157,"tags":2158,"stars":2127,"repoUrl":2128,"updatedAt":2166},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2159,2162,2163],{"name":2160,"slug":2161,"type":15},"Debugging","debugging",{"name":2150,"slug":2151,"type":15},{"name":2164,"slug":2165,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2168,"name":2168,"fn":2169,"description":2170,"org":2171,"tags":2172,"stars":2127,"repoUrl":2128,"updatedAt":2179},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2173,2176],{"name":2174,"slug":2175,"type":15},"Best Practices","best-practices",{"name":2177,"slug":2178,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2181,"name":2181,"fn":2182,"description":2183,"org":2184,"tags":2185,"stars":2127,"repoUrl":2128,"updatedAt":2193},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2186,2189,2192],{"name":2187,"slug":2188,"type":15},"Machine Learning","machine-learning",{"name":2190,"slug":2191,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2195,"name":2195,"fn":2196,"description":2197,"org":2198,"tags":2199,"stars":2127,"repoUrl":2128,"updatedAt":2204},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2200,2203],{"name":2201,"slug":2202,"type":15},"QA","qa",{"name":2046,"slug":2047,"type":15},"2026-07-14T05:25:53.673039",{"slug":2206,"name":2206,"fn":2207,"description":2208,"org":2209,"tags":2210,"stars":2127,"repoUrl":2128,"updatedAt":2213},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2211,2212],{"name":2016,"slug":2017,"type":15},{"name":2019,"slug":2020,"type":15},"2026-07-14T05:25:49.362534",{"slug":2215,"name":2215,"fn":2216,"description":2217,"org":2218,"tags":2219,"stars":2127,"repoUrl":2128,"updatedAt":2227},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2220,2223,2224],{"name":2221,"slug":2222,"type":15},"Code Review","code-review",{"name":2150,"slug":2151,"type":15},{"name":2225,"slug":2226,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2229,"name":2229,"fn":2230,"description":2231,"org":2232,"tags":2233,"stars":2127,"repoUrl":2128,"updatedAt":2236},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2234,2235],{"name":2201,"slug":2202,"type":15},{"name":2046,"slug":2047,"type":15},"2026-07-14T05:25:54.928983",{"slug":2238,"name":2238,"fn":2239,"description":2240,"org":2241,"tags":2242,"stars":2127,"repoUrl":2128,"updatedAt":2245},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2243,2244],{"name":2056,"slug":2057,"type":15},{"name":2137,"slug":2138,"type":15},"2026-07-30T05:29:03.275638",496]