[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-cupynumeric-parallel-data-load":3,"mdc--t3wkq8-key":37,"related-org-nvidia-cupynumeric-parallel-data-load":2590,"related-repo-nvidia-cupynumeric-parallel-data-load":2750},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"cupynumeric-parallel-data-load","load distributed datasets into cuPyNumeric","Load a sharded, on-disk dataset (sharded .npy, Parquet\u002FArrow, raw binary, sharded HDF5, custom layouts) into a distributed cuPyNumeric ndarray via a manual partition + leaf @task launch with CPU\u002FOMP\u002FGPU variants. Use when no single-call loader fits, including when per-shard row counts differ across files. Prefer cupynumeric.load or legate.io.hdf5.from_file when they apply.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Datasets","datasets",{"name":20,"slug":21,"type":15},"Data Engineering","data-engineering",{"name":23,"slug":24,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:29:02.93907","CC-BY-4.0 OR Apache-2.0",281,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fcupynumeric-parallel-data-load","---\nname: cupynumeric-parallel-data-load\ndescription: Load a sharded, on-disk dataset (sharded .npy, Parquet\u002FArrow, raw binary, sharded HDF5, custom layouts) into a distributed cuPyNumeric ndarray via a manual partition + leaf @task launch with CPU\u002FOMP\u002FGPU variants. Use when no single-call loader fits, including when per-shard row counts differ across files. Prefer cupynumeric.load or legate.io.hdf5.from_file when they apply.\nlicense: CC-BY-4.0 OR Apache-2.0\ncompatibility: linux-x86_64, linux-aarch64, darwin-aarch64, wsl-x86_64\nmetadata:\n version: \"1.0.0\"\n author: \"NVIDIA Corporation \u003Clegate@nvidia.com>\"\n upstream: https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fcupynumeric\n docs: https:\u002F\u002Fdocs.nvidia.com\u002Fcupynumeric\u002Flatest\u002F\n tags:\n - cupynumeric\n - legate\n - data-loading\n - io\n - distributed\n - parallel\n - gpu\n - sharded-data\n---\n\n# Parallel sharded data -> cupynumeric load\n\n**Why this skill exists.** cupynumeric mirrors NumPy's array API,\nincluding `cupynumeric.load` for a single `.npy` file. Beyond that,\nfile *loading* lives in Legate, not cupynumeric:\n\n| Format | Built-in loader |\n|---|---|\n| Single `.npy` | `cupynumeric.load(path)` (NumPy-API parity) |\n| HDF5 (single file) | `legate.io.hdf5.from_file` \u002F `from_file_batched` |\n| Sharded multi-file (any format), Parquet\u002FArrow, raw binary, custom layouts | **No built-in loader — this skill.** |\n\nThis skill shows the canonical way to fill the gap in the last row:\nwrite a Legate Python task that calls the third-party reader the\nformat needs (`h5py`, `pyarrow`, `np.memmap`, ...) inside the\ntask body, and let Legate distribute the reads across GPUs \u002F nodes.\nFor the formats with a built-in loader, prefer it unless you need a\ncustom in-task body (mmap-based loader, format-specific decoder,\nsidecar metadata, partial \u002F sharded reads).\n\nCanonical pattern: **manual partition + manual task launch, sized to\nthe machine, not the files.** Only axis 0 is sharded; trailing axes\nride along inside each tile. Per-shard row counts may differ across\nfiles (only `dtype` and trailing axes must match); the launch fills\nevery available processor regardless of how many files there are.\n\n`.npy` is the worked example because the header carries shape and\ndtype on disk, but the skeleton applies to any format with cheap\nrange\u002Fslice reads (raw binary, HDF5, Parquet\u002FArrow — see \"Other\nformats\" below). Reference implementation:\n[`assets\u002Fexamples\u002Fparallel_npy_load.py`](assets\u002Fexamples\u002Fparallel_npy_load.py).\n\n## Data layout assumption\n\nThis skill is purely about **loading** — it assumes the data is already\nlaid out on a shared filesystem in some predictable, indexable way.\nProducing those files is out of scope (the example ships a `write`\nsubcommand for convenience, but real users bring their own).\n\nThe worked example assumes one specific layout:\n\n- A directory containing files named `shard_0000.npy`, `shard_0001.npy`,\n ... in a contiguous integer sequence (zero-padded width 4).\n- All shards share the same `dtype` and the same trailing axes\n (`shape[1:]`); **axis 0 (rows per shard) may differ across files** —\n the recipe builds a cumulative row-offset table and reads each\n file's overlapping slice from inside the leaf task.\n- The directory is visible to every rank (shared filesystem for\n multi-node runs).\n\nThe example's `discover_layout()` prints what it found and hard-fails\nwith a descriptive error when the layout is wrong (missing directory,\nno shards, mismatched `dtype` \u002F trailing axes, or a hole in the\ncontiguous `shard_NNNN.npy` sequence).\n\nIf your data lives in a different layout — fixed-stride raw binary, an\nHDF5 file with one dataset per shard, a directory tree, ... — only the\nglob pattern, the per-file reader (step 4 below), and the metadata\ndiscovery (step 1 below) change. The partitioning and launch machinery\nis layout-agnostic.\n\n## When to use\n\nSee the format table above for the routing decision (built-in loader\nvs. this skill). Beyond that, two additional cues that this skill is\nthe right fit:\n\n- Replacing sequential `np.concatenate([read(f) for f in files])` with\n parallel per-GPU reads.\n- Demonstrating how a user-defined Legate Python task writes into a\n cupynumeric output array via a manual launch.\n\n## Examples\n\nPaths below are written relative to this skill's directory (the script\nships at `assets\u002Fexamples\u002Fparallel_npy_load.py`). Adjust the prefix to\nmatch wherever your skill is installed (e.g.\n`skills\u002Fcupynumeric-parallel-data-load\u002Fassets\u002F...` if the skill lives\nunder a top-level `skills\u002F` directory).\n\n```bash\n# Single-node, 4 GPUs.\nlegate --gpus 4 --fbmem 4000 --min-gpu-chunk 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n read --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n```\n\n```bash\n# Multi-node, 2 nodes x 4 GPUs (slurm), shared filesystem at --shard-dir.\n# Generate the shards once on rank 0, then re-run `read` at any scale.\nlegate --launcher srun --nodes 2 --cpus 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n write --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n\nlegate --launcher srun --nodes 2 --ranks-per-node 4 \\\n --gpus 4 --fbmem 4000 --min-gpu-chunk 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n read --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n```\n\nNo layout flags — the read driver walks every `.npy` header to recover\nper-file row counts, the trailing shape, and the dtype, then derives\n`tile_rows` from the available processor count.\n\n`--min-gpu-chunk 1` is only needed when the per-tile element count is\nbelow Legate's default minimum chunk size for GPU launches (e.g. the\nworked example's defaults — total rows split across 4 GPUs at\n`~1M` per tile — fall below the threshold and would otherwise be\nfolded onto a single GPU). For production-sized datasets (tens of\nmillions of elements per tile or larger) you can drop the flag and\nlet Legate use its default. Bumping it to a moderate value (e.g.\n`--min-gpu-chunk 1024`) is fine when each tile is large enough that\nper-task overhead matters more than getting *every* GPU a tile.\n\n## Instructions\n\nFive steps from a `.npy` worked example; only step 1 (parsing the\nformat header) and step 4 (the per-file reader inside the task body)\nare format-specific. The other three (allocate destination, partition,\nfence) are reused unchanged across formats — see \"Other formats\" below\nfor the swap-points.\n\n### 1. Read the metadata from every shard\n\nScan the directory and peek at every `.npy` header (`mmap_mode=\"r\"`\nreads only the header). The header carries the per-shard shape and\ndtype, so the driver can recover total rows, trailing shape, and a\ncumulative row-offset table without ever loading the data:\n\n```python\npaths = sorted(SHARD_DIR.glob(\"shard_*.npy\"))\n\nper_file_rows = [] # rows along axis 0 per file\ntrailing_shape = None # shape[1:], must match across files\ndtype = None\nfor p in paths:\n hdr = np.load(p, mmap_mode=\"r\")\n if trailing_shape is None:\n trailing_shape = tuple(hdr.shape[1:])\n dtype = hdr.dtype\n elif tuple(hdr.shape[1:]) != trailing_shape or hdr.dtype != dtype:\n raise RuntimeError(\n f\"{p.name}: trailing shape \u002F dtype mismatch \"\n f\"({hdr.shape[1:]}\u002F{hdr.dtype} vs {trailing_shape}\u002F{dtype})\"\n )\n per_file_rows.append(int(hdr.shape[0]))\n\ncum_rows = np.cumsum([0] + per_file_rows, dtype=np.int64) # length N+1\ntotal_rows = int(cum_rows[-1])\n```\n\nThe snippet above enforces matching `dtype` and `trailing_shape` (i.e.\n`shape[1:]`) across files. **Per-shard row counts may differ** — the\ncum-rows table handles that. Production code should also verify that\nnames form a contiguous `shard_0000.npy ... shard_NNNN.npy` sequence\n(omitted from the snippet for brevity; see `discover_layout()` in the\nworked example). Discovery relies only on what the\non-disk format itself exposes (the `.npy` header here, `.shape` \u002F\n`.dtype` for HDF5, etc.); any sidecar (manifest, content hashes) is a\nseparate verification step on top.\n\n### 2. Create the cupynumeric output store from the metadata\n\nThe total array spans `total_rows` along axis 0; trailing axes come\nfrom `trailing_shape` unchanged. Use `cn.empty` — the task overwrites\nevery cell, zero-init would be wasted.\n\n```python\nimport cupynumeric as cn\n\ntotal_shape = (total_rows,) + trailing_shape\nout = cn.empty(total_shape, dtype=dtype)\n```\n\n### 3. Tile the store by processor count\n\nThe launch shape is sized to the **available processors**, not to the\nfile count. Pick `tile_rows = ceil(total_rows \u002F num_processors)` and\npartition axis 0 by that tile size. Trailing axes are not partitioned\n(tile spans the full extent there). The last tile is allowed to be\nshort — that's exactly what `partition_by_tiling` supports — so the\nrecipe needs no divisibility constraint.\n\n```python\nfrom legate.core import TaskTarget, get_legate_runtime\nfrom legate.core.data_interface import as_logical_array\n\nruntime = get_legate_runtime()\nmachine = runtime.get_machine()\nnum_processors = max(\n machine.count(TaskTarget.GPU),\n machine.count(TaskTarget.OMP),\n machine.count(TaskTarget.CPU),\n 1,\n)\n\ntile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\ntile_shape = (tile_rows,) + trailing_shape\npartition = as_logical_array(out).data.partition_by_tiling(tile_shape)\n\nnum_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows # match partition tile count\n```\n\n### 4. Define the leaf task and launch it manually\n\n`PATHS` and `CUM_ROWS` (the file paths and cumulative row-offset\ntable from step 1) plus `TILE_ROWS` are populated as module globals\nby the driver before launching; control replication runs the driver\non every rank, so every worker sees identical values.\n\nEach task builds its consumer view first (cupy on GPU, numpy on\nCPU\u002FOMP) and reads the tile's actual row count from `view.shape[0]`\n— `PhysicalStore` itself has no `.shape` attribute, so going through\nthe view is required. It then computes its global row range from its\nlaunch coordinate and that row count, bisects `cum_rows` for the\noverlapping file(s), and copies each overlapping file slice into the\nmatching destination slice. Register CPU, OMP, and GPU variants so\nthe same launch runs unchanged anywhere; dispatch on\n`ctx.get_variant_kind()` picks the consumer matching where the\n`OutputStore` is resident (`cp.from_dlpack(dst)` for FBMEM,\n`np.asarray(dst)` for SYSMEM). cupy is imported inside the GPU\nbranch only, so the task body loads on machines without cupy.\n\n```python\nimport bisect\nfrom legate.core import TaskContext, VariantCode\nfrom legate.core.task import OutputStore, task\n\n@task(variants=(VariantCode.CPU, VariantCode.OMP, VariantCode.GPU))\ndef load_tile(ctx: TaskContext, dst: OutputStore) -> None:\n t = ctx.task_index[0] # tile index 0..num_tasks-1\n\n variant = ctx.get_variant_kind()\n if variant == VariantCode.GPU:\n import cupy as cp # lazy: only on GPU\n view = cp.from_dlpack(dst)\n else:\n view = np.asarray(dst) # zero-copy numpy view\n\n tile_rows_actual = view.shape[0] # short on the last tile\n row_start = t * TILE_ROWS # global axis-0 start\n row_end = row_start + tile_rows_actual\n\n # Find the half-open range of file indices that overlap [row_start, row_end).\n first_file = bisect.bisect_right(CUM_ROWS, row_start) - 1\n last_file = bisect.bisect_right(CUM_ROWS, row_end - 1) - 1\n\n for f in range(first_file, last_file + 1):\n # Intersection of tile [row_start, row_end) with file [cum[f], cum[f+1]).\n lo = max(row_start, int(CUM_ROWS[f]))\n hi = min(row_end, int(CUM_ROWS[f + 1]))\n file_lo = lo - int(CUM_ROWS[f])\n file_hi = hi - int(CUM_ROWS[f])\n dst_lo = lo - row_start\n dst_hi = hi - row_start\n chunk = np.ascontiguousarray(\n np.load(PATHS[f], mmap_mode=\"r\")[file_lo:file_hi]\n )\n if variant == VariantCode.GPU:\n view[dst_lo:dst_hi].set(chunk) # cudaMemcpyAsync H2D\n else:\n view[dst_lo:dst_hi] = chunk # zero-copy numpy write\n\nmanual_task = runtime.create_manual_task(\n load_tile.library,\n load_tile.task_id,\n (num_tasks,), # launch domain == tile count\n)\nmanual_task.add_output(partition)\nmanual_task.execute()\n```\n\nBoth consumers go through `PhysicalStore`'s native producers\n(`__dlpack__` for cupy, `__array_interface__` for `np.asarray`) —\nzero-copy views of the local tile. Bisect cost is `O(log num_shards)`\nand the inner loop typically iterates 1–2 times (tiles overlap at\nmost a couple of files).\n\n### 5. Fence and verify\n\n```python\nget_legate_runtime().issue_execution_fence(block=True)\n```\n\n## Hard constraints\n\n1. **All shards must share `dtype` and trailing axes (`shape[1:]`).**\n The recipe stacks shards along axis 0; the destination's trailing\n axes come from `trailing_shape`, which the discovery step locks to\n the value of the first file. Per-shard row counts (`shape[0]`) may\n freely differ — the cumulative-offset table handles them. The\n example rejects any shard whose `dtype` or trailing shape differs\n from the first one with a descriptive error.\n\n2. **Pick the consumer that matches the variant.** `cp.from_dlpack`\n rejects SYSMEM-resident stores; `np.asarray` silently returns a\n host view of an FBMEM-resident store you can't actually write\n through. Dispatch on `ctx.get_variant_kind()` so each variant uses\n its own consumer — see step 4.\n\n3. **mmap views aren't always C-contiguous** — wrap each per-file\n slice with `np.ascontiguousarray(arr[file_lo:file_hi])` before\n `.set()` or the numpy in-place write.\n\n4. **Multi-node: `SHARD_DIR` must be on a shared filesystem.** Every\n worker (on every rank) opens shards by path; node-local `\u002Ftmp` paths\n only work for single-node demos.\n\n## Variants\n\n### Uniform-shard fast path (one task per file)\n\nWhen every shard already has the same `(shape, dtype)` and you happen\nto have `num_shards` processors available, the cum-rows \u002F bisect\nmachinery is overhead. Set `tile_rows = shard_shape[0]` and\n`num_tasks = num_shards`; the partition then has one tile per file\nand each task reads exactly one file end-to-end (no bisect, no inner\nloop). The driver-side switch is a one-liner:\n\n```python\nif all(r == per_file_rows[0] for r in per_file_rows) and num_shards == num_processors:\n tile_rows = per_file_rows[0]\nelse:\n tile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\n```\n\nThe same `load_tile` task body still works in either mode — the inner\nloop just happens to iterate exactly once per task. There's no need\nfor a separate task body for the fast path.\n\n### Over-decompose for better load balancing\n\nThe default `tile_rows = ceil(total_rows \u002F num_processors)` gives one\ntile per processor. To over-decompose by a factor `K` (smaller tiles,\nmore point tasks, finer-grained queueing), divide by `K * num_processors`\ninstead:\n\n```python\ntile_rows = max(1, (total_rows + K * num_processors - 1) \u002F\u002F (K * num_processors))\n```\n\n`num_tasks = ceil(total_rows \u002F tile_rows)` then expands to roughly\n`K * num_processors`. The same task body still works — bisect just lands\non more tasks per file.\n\n### Other formats\n\nOnly the per-file reader inside `load_tile` changes. The reader's\ncontract: given a file path and a half-open row range\n`[file_lo, file_hi)` along axis 0, return a numpy array of shape\n`(file_hi - file_lo,) + trailing_shape` that can be made C-contiguous.\nCheap range\u002Fslice reads are required — formats that only support\n\"read the whole file\" defeat the partial-overlap case (a tile that\ncovers only part of one file).\n\n| Format | Reader inside the leaf task |\n|---|---|\n| **`.npy`** (worked example) | `host = np.ascontiguousarray(np.load(p, mmap_mode=\"r\")[file_lo:file_hi])` |\n| **Raw binary** (fixed-shape) | `arr = np.memmap(p, dtype=DTYPE, mode=\"r\", shape=(rows_in_file, *trailing_shape)); host = np.ascontiguousarray(arr[file_lo:file_hi])` |\n| **HDF5** | `with h5py.File(p, \"r\") as f: host = np.ascontiguousarray(f[\"data\"][file_lo:file_hi])` |\n| **Parquet \u002F Arrow** | `tbl = pq.read_table(p, columns=..., use_threads=False).slice(file_lo, file_hi - file_lo); host = tbl.to_pandas().values` |\n\n(For built-in single-call loaders per format, see the \"Why this skill\nexists\" table at the top of this file.)\n\nThe discovery step (step 1) parses each format's metadata: `.npy` \u002F\nHDF5 \u002F Parquet all carry per-file row count + dtype on disk.\nRaw binary doesn't — sidecar or derive from file size.\n\n## Common pitfalls\n\n### `cn.asarray(dst)` is illegal in a leaf task\n\nInside a `@task` body, any cupynumeric op that touches the top-level\nruntime — `cn.asarray(store)`, slice assignment `cn_dst[s] = host_np` —\ntriggers `create_index_space` from the wrong context and Legion aborts:\n\n```\nLEGION API USAGE EXCEPTION: Invalid task context passed to runtime call\ncreate_index_space\n```\n\nFix: consume the DLPack capsule with a **third-party** library (cupy \u002F\ntorch \u002F numpy) inside leaf tasks. `cn.asarray` is fine in the driver,\njust not in leaf tasks. See `examples\u002Fdlpack\u002Fleaf_task_interop.py` for\nthe torch-flavoured workaround.\n\n### In-task `assert` aborts the runtime\n\nLegate treats unraised exceptions in a `@task` as a contract violation\nand aborts unless the task was registered with `throws_exception()`.\nSanity-check on the host before launching.\n\n### Launch domain must match the partition tile count\n\n`create_manual_task(launch_shape=...)` and `partition_by_tiling(...)`\nare independent — the runtime doesn't catch a mismatch. Larger launch\ndomain → out-of-range tiles; smaller → unwritten tiles. Always derive\nboth from the same `(total_rows, tile_rows)` via two separate `ceil`\ndivisions (sizing the launch domain to `num_processors` directly\nwould over-launch when `num_processors > total_rows`):\n\n```python\ntile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\nnum_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows\npartition = ...partition_by_tiling((tile_rows,) + trailing_shape)\nruntime.create_manual_task(load_tile.library, load_tile.task_id, (num_tasks,))\n```\n",{"data":38,"body":54},{"name":4,"description":6,"license":29,"compatibility":39,"metadata":40},"linux-x86_64, linux-aarch64, darwin-aarch64, wsl-x86_64",{"version":41,"author":42,"upstream":43,"docs":44,"tags":45},"1.0.0","NVIDIA Corporation \u003Clegate@nvidia.com>","https:\u002F\u002Fgithub.com\u002Fnv-legate\u002Fcupynumeric","https:\u002F\u002Fdocs.nvidia.com\u002Fcupynumeric\u002Flatest\u002F",[46,47,48,49,50,51,52,53],"cupynumeric","legate","data-loading","io","distributed","parallel","gpu","sharded-data",{"type":55,"children":56},"root",[57,66,103,196,224,244,266,273,292,297,354,382,387,393,398,419,425,453,555,754,774,808,814,826,833,853,1021,1093,1099,1127,1165,1171,1199,1339,1345,1371,1439,1833,1877,1883,1897,1903,2037,2043,2049,2086,2125,2138,2144,2172,2186,2204,2210,2238,2346,2351,2363,2369,2381,2418,2428,2456,2470,2490,2496,2546,2584],{"type":58,"tag":59,"props":60,"children":62},"element","h1",{"id":61},"parallel-sharded-data-cupynumeric-load",[63],{"type":64,"value":65},"text","Parallel sharded data -> cupynumeric load",{"type":58,"tag":67,"props":68,"children":69},"p",{},[70,76,78,85,87,93,95,101],{"type":58,"tag":71,"props":72,"children":73},"strong",{},[74],{"type":64,"value":75},"Why this skill exists.",{"type":64,"value":77}," cupynumeric mirrors NumPy's array API,\nincluding ",{"type":58,"tag":79,"props":80,"children":82},"code",{"className":81},[],[83],{"type":64,"value":84},"cupynumeric.load",{"type":64,"value":86}," for a single ",{"type":58,"tag":79,"props":88,"children":90},{"className":89},[],[91],{"type":64,"value":92},".npy",{"type":64,"value":94}," file. Beyond that,\nfile ",{"type":58,"tag":96,"props":97,"children":98},"em",{},[99],{"type":64,"value":100},"loading",{"type":64,"value":102}," lives in Legate, not cupynumeric:",{"type":58,"tag":104,"props":105,"children":106},"table",{},[107,126],{"type":58,"tag":108,"props":109,"children":110},"thead",{},[111],{"type":58,"tag":112,"props":113,"children":114},"tr",{},[115,121],{"type":58,"tag":116,"props":117,"children":118},"th",{},[119],{"type":64,"value":120},"Format",{"type":58,"tag":116,"props":122,"children":123},{},[124],{"type":64,"value":125},"Built-in loader",{"type":58,"tag":127,"props":128,"children":129},"tbody",{},[130,155,180],{"type":58,"tag":112,"props":131,"children":132},{},[133,144],{"type":58,"tag":134,"props":135,"children":136},"td",{},[137,139],{"type":64,"value":138},"Single ",{"type":58,"tag":79,"props":140,"children":142},{"className":141},[],[143],{"type":64,"value":92},{"type":58,"tag":134,"props":145,"children":146},{},[147,153],{"type":58,"tag":79,"props":148,"children":150},{"className":149},[],[151],{"type":64,"value":152},"cupynumeric.load(path)",{"type":64,"value":154}," (NumPy-API parity)",{"type":58,"tag":112,"props":156,"children":157},{},[158,163],{"type":58,"tag":134,"props":159,"children":160},{},[161],{"type":64,"value":162},"HDF5 (single file)",{"type":58,"tag":134,"props":164,"children":165},{},[166,172,174],{"type":58,"tag":79,"props":167,"children":169},{"className":168},[],[170],{"type":64,"value":171},"legate.io.hdf5.from_file",{"type":64,"value":173}," \u002F ",{"type":58,"tag":79,"props":175,"children":177},{"className":176},[],[178],{"type":64,"value":179},"from_file_batched",{"type":58,"tag":112,"props":181,"children":182},{},[183,188],{"type":58,"tag":134,"props":184,"children":185},{},[186],{"type":64,"value":187},"Sharded multi-file (any format), Parquet\u002FArrow, raw binary, custom layouts",{"type":58,"tag":134,"props":189,"children":190},{},[191],{"type":58,"tag":71,"props":192,"children":193},{},[194],{"type":64,"value":195},"No built-in loader — this skill.",{"type":58,"tag":67,"props":197,"children":198},{},[199,201,207,209,215,216,222],{"type":64,"value":200},"This skill shows the canonical way to fill the gap in the last row:\nwrite a Legate Python task that calls the third-party reader the\nformat needs (",{"type":58,"tag":79,"props":202,"children":204},{"className":203},[],[205],{"type":64,"value":206},"h5py",{"type":64,"value":208},", ",{"type":58,"tag":79,"props":210,"children":212},{"className":211},[],[213],{"type":64,"value":214},"pyarrow",{"type":64,"value":208},{"type":58,"tag":79,"props":217,"children":219},{"className":218},[],[220],{"type":64,"value":221},"np.memmap",{"type":64,"value":223},", ...) inside the\ntask body, and let Legate distribute the reads across GPUs \u002F nodes.\nFor the formats with a built-in loader, prefer it unless you need a\ncustom in-task body (mmap-based loader, format-specific decoder,\nsidecar metadata, partial \u002F sharded reads).",{"type":58,"tag":67,"props":225,"children":226},{},[227,229,234,236,242],{"type":64,"value":228},"Canonical pattern: ",{"type":58,"tag":71,"props":230,"children":231},{},[232],{"type":64,"value":233},"manual partition + manual task launch, sized to\nthe machine, not the files.",{"type":64,"value":235}," Only axis 0 is sharded; trailing axes\nride along inside each tile. Per-shard row counts may differ across\nfiles (only ",{"type":58,"tag":79,"props":237,"children":239},{"className":238},[],[240],{"type":64,"value":241},"dtype",{"type":64,"value":243}," and trailing axes must match); the launch fills\nevery available processor regardless of how many files there are.",{"type":58,"tag":67,"props":245,"children":246},{},[247,252,254,264],{"type":58,"tag":79,"props":248,"children":250},{"className":249},[],[251],{"type":64,"value":92},{"type":64,"value":253}," is the worked example because the header carries shape and\ndtype on disk, but the skeleton applies to any format with cheap\nrange\u002Fslice reads (raw binary, HDF5, Parquet\u002FArrow — see \"Other\nformats\" below). Reference implementation:\n",{"type":58,"tag":255,"props":256,"children":258},"a",{"href":257},"assets\u002Fexamples\u002Fparallel_npy_load.py",[259],{"type":58,"tag":79,"props":260,"children":262},{"className":261},[],[263],{"type":64,"value":257},{"type":64,"value":265},".",{"type":58,"tag":267,"props":268,"children":270},"h2",{"id":269},"data-layout-assumption",[271],{"type":64,"value":272},"Data layout assumption",{"type":58,"tag":67,"props":274,"children":275},{},[276,278,282,284,290],{"type":64,"value":277},"This skill is purely about ",{"type":58,"tag":71,"props":279,"children":280},{},[281],{"type":64,"value":100},{"type":64,"value":283}," — it assumes the data is already\nlaid out on a shared filesystem in some predictable, indexable way.\nProducing those files is out of scope (the example ships a ",{"type":58,"tag":79,"props":285,"children":287},{"className":286},[],[288],{"type":64,"value":289},"write",{"type":64,"value":291},"\nsubcommand for convenience, but real users bring their own).",{"type":58,"tag":67,"props":293,"children":294},{},[295],{"type":64,"value":296},"The worked example assumes one specific layout:",{"type":58,"tag":298,"props":299,"children":300},"ul",{},[301,322,349],{"type":58,"tag":302,"props":303,"children":304},"li",{},[305,307,313,314,320],{"type":64,"value":306},"A directory containing files named ",{"type":58,"tag":79,"props":308,"children":310},{"className":309},[],[311],{"type":64,"value":312},"shard_0000.npy",{"type":64,"value":208},{"type":58,"tag":79,"props":315,"children":317},{"className":316},[],[318],{"type":64,"value":319},"shard_0001.npy",{"type":64,"value":321},",\n... in a contiguous integer sequence (zero-padded width 4).",{"type":58,"tag":302,"props":323,"children":324},{},[325,327,332,334,340,342,347],{"type":64,"value":326},"All shards share the same ",{"type":58,"tag":79,"props":328,"children":330},{"className":329},[],[331],{"type":64,"value":241},{"type":64,"value":333}," and the same trailing axes\n(",{"type":58,"tag":79,"props":335,"children":337},{"className":336},[],[338],{"type":64,"value":339},"shape[1:]",{"type":64,"value":341},"); ",{"type":58,"tag":71,"props":343,"children":344},{},[345],{"type":64,"value":346},"axis 0 (rows per shard) may differ across files",{"type":64,"value":348}," —\nthe recipe builds a cumulative row-offset table and reads each\nfile's overlapping slice from inside the leaf task.",{"type":58,"tag":302,"props":350,"children":351},{},[352],{"type":64,"value":353},"The directory is visible to every rank (shared filesystem for\nmulti-node runs).",{"type":58,"tag":67,"props":355,"children":356},{},[357,359,365,367,372,374,380],{"type":64,"value":358},"The example's ",{"type":58,"tag":79,"props":360,"children":362},{"className":361},[],[363],{"type":64,"value":364},"discover_layout()",{"type":64,"value":366}," prints what it found and hard-fails\nwith a descriptive error when the layout is wrong (missing directory,\nno shards, mismatched ",{"type":58,"tag":79,"props":368,"children":370},{"className":369},[],[371],{"type":64,"value":241},{"type":64,"value":373}," \u002F trailing axes, or a hole in the\ncontiguous ",{"type":58,"tag":79,"props":375,"children":377},{"className":376},[],[378],{"type":64,"value":379},"shard_NNNN.npy",{"type":64,"value":381}," sequence).",{"type":58,"tag":67,"props":383,"children":384},{},[385],{"type":64,"value":386},"If your data lives in a different layout — fixed-stride raw binary, an\nHDF5 file with one dataset per shard, a directory tree, ... — only the\nglob pattern, the per-file reader (step 4 below), and the metadata\ndiscovery (step 1 below) change. The partitioning and launch machinery\nis layout-agnostic.",{"type":58,"tag":267,"props":388,"children":390},{"id":389},"when-to-use",[391],{"type":64,"value":392},"When to use",{"type":58,"tag":67,"props":394,"children":395},{},[396],{"type":64,"value":397},"See the format table above for the routing decision (built-in loader\nvs. this skill). Beyond that, two additional cues that this skill is\nthe right fit:",{"type":58,"tag":298,"props":399,"children":400},{},[401,414],{"type":58,"tag":302,"props":402,"children":403},{},[404,406,412],{"type":64,"value":405},"Replacing sequential ",{"type":58,"tag":79,"props":407,"children":409},{"className":408},[],[410],{"type":64,"value":411},"np.concatenate([read(f) for f in files])",{"type":64,"value":413}," with\nparallel per-GPU reads.",{"type":58,"tag":302,"props":415,"children":416},{},[417],{"type":64,"value":418},"Demonstrating how a user-defined Legate Python task writes into a\ncupynumeric output array via a manual launch.",{"type":58,"tag":267,"props":420,"children":422},{"id":421},"examples",[423],{"type":64,"value":424},"Examples",{"type":58,"tag":67,"props":426,"children":427},{},[428,430,435,437,443,445,451],{"type":64,"value":429},"Paths below are written relative to this skill's directory (the script\nships at ",{"type":58,"tag":79,"props":431,"children":433},{"className":432},[],[434],{"type":64,"value":257},{"type":64,"value":436},"). Adjust the prefix to\nmatch wherever your skill is installed (e.g.\n",{"type":58,"tag":79,"props":438,"children":440},{"className":439},[],[441],{"type":64,"value":442},"skills\u002Fcupynumeric-parallel-data-load\u002Fassets\u002F...",{"type":64,"value":444}," if the skill lives\nunder a top-level ",{"type":58,"tag":79,"props":446,"children":448},{"className":447},[],[449],{"type":64,"value":450},"skills\u002F",{"type":64,"value":452}," directory).",{"type":58,"tag":454,"props":455,"children":460},"pre",{"className":456,"code":457,"language":458,"meta":459,"style":459},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Single-node, 4 GPUs.\nlegate --gpus 4 --fbmem 4000 --min-gpu-chunk 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n read --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n","bash","",[461],{"type":58,"tag":79,"props":462,"children":463},{"__ignoreMap":459},[464,476,523,536],{"type":58,"tag":465,"props":466,"children":469},"span",{"class":467,"line":468},"line",1,[470],{"type":58,"tag":465,"props":471,"children":473},{"style":472},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[474],{"type":64,"value":475},"# Single-node, 4 GPUs.\n",{"type":58,"tag":465,"props":477,"children":479},{"class":467,"line":478},2,[480,485,491,497,502,507,512,517],{"type":58,"tag":465,"props":481,"children":483},{"style":482},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[484],{"type":64,"value":47},{"type":58,"tag":465,"props":486,"children":488},{"style":487},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[489],{"type":64,"value":490}," --gpus",{"type":58,"tag":465,"props":492,"children":494},{"style":493},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[495],{"type":64,"value":496}," 4",{"type":58,"tag":465,"props":498,"children":499},{"style":487},[500],{"type":64,"value":501}," --fbmem",{"type":58,"tag":465,"props":503,"children":504},{"style":493},[505],{"type":64,"value":506}," 4000",{"type":58,"tag":465,"props":508,"children":509},{"style":487},[510],{"type":64,"value":511}," --min-gpu-chunk",{"type":58,"tag":465,"props":513,"children":514},{"style":493},[515],{"type":64,"value":516}," 1",{"type":58,"tag":465,"props":518,"children":520},{"style":519},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[521],{"type":64,"value":522}," \\\n",{"type":58,"tag":465,"props":524,"children":526},{"class":467,"line":525},3,[527,532],{"type":58,"tag":465,"props":528,"children":529},{"style":487},[530],{"type":64,"value":531}," assets\u002Fexamples\u002Fparallel_npy_load.py",{"type":58,"tag":465,"props":533,"children":534},{"style":519},[535],{"type":64,"value":522},{"type":58,"tag":465,"props":537,"children":539},{"class":467,"line":538},4,[540,545,550],{"type":58,"tag":465,"props":541,"children":542},{"style":487},[543],{"type":64,"value":544}," read",{"type":58,"tag":465,"props":546,"children":547},{"style":487},[548],{"type":64,"value":549}," --shard-dir",{"type":58,"tag":465,"props":551,"children":552},{"style":487},[553],{"type":64,"value":554}," \u002Fshared\u002Fscratch\u002Fdemo\n",{"type":58,"tag":454,"props":556,"children":558},{"className":456,"code":557,"language":458,"meta":459,"style":459},"# Multi-node, 2 nodes x 4 GPUs (slurm), shared filesystem at --shard-dir.\n# Generate the shards once on rank 0, then re-run `read` at any scale.\nlegate --launcher srun --nodes 2 --cpus 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n write --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n\nlegate --launcher srun --nodes 2 --ranks-per-node 4 \\\n --gpus 4 --fbmem 4000 --min-gpu-chunk 1 \\\n assets\u002Fexamples\u002Fparallel_npy_load.py \\\n read --shard-dir \u002Fshared\u002Fscratch\u002Fdemo\n",[559],{"type":58,"tag":79,"props":560,"children":561},{"__ignoreMap":459},[562,570,578,618,629,646,656,693,726,738],{"type":58,"tag":465,"props":563,"children":564},{"class":467,"line":468},[565],{"type":58,"tag":465,"props":566,"children":567},{"style":472},[568],{"type":64,"value":569},"# Multi-node, 2 nodes x 4 GPUs (slurm), shared filesystem at --shard-dir.\n",{"type":58,"tag":465,"props":571,"children":572},{"class":467,"line":478},[573],{"type":58,"tag":465,"props":574,"children":575},{"style":472},[576],{"type":64,"value":577},"# Generate the shards once on rank 0, then re-run `read` at any scale.\n",{"type":58,"tag":465,"props":579,"children":580},{"class":467,"line":525},[581,585,590,595,600,605,610,614],{"type":58,"tag":465,"props":582,"children":583},{"style":482},[584],{"type":64,"value":47},{"type":58,"tag":465,"props":586,"children":587},{"style":487},[588],{"type":64,"value":589}," --launcher",{"type":58,"tag":465,"props":591,"children":592},{"style":487},[593],{"type":64,"value":594}," srun",{"type":58,"tag":465,"props":596,"children":597},{"style":487},[598],{"type":64,"value":599}," --nodes",{"type":58,"tag":465,"props":601,"children":602},{"style":493},[603],{"type":64,"value":604}," 2",{"type":58,"tag":465,"props":606,"children":607},{"style":487},[608],{"type":64,"value":609}," --cpus",{"type":58,"tag":465,"props":611,"children":612},{"style":493},[613],{"type":64,"value":516},{"type":58,"tag":465,"props":615,"children":616},{"style":519},[617],{"type":64,"value":522},{"type":58,"tag":465,"props":619,"children":620},{"class":467,"line":538},[621,625],{"type":58,"tag":465,"props":622,"children":623},{"style":487},[624],{"type":64,"value":531},{"type":58,"tag":465,"props":626,"children":627},{"style":519},[628],{"type":64,"value":522},{"type":58,"tag":465,"props":630,"children":632},{"class":467,"line":631},5,[633,638,642],{"type":58,"tag":465,"props":634,"children":635},{"style":487},[636],{"type":64,"value":637}," write",{"type":58,"tag":465,"props":639,"children":640},{"style":487},[641],{"type":64,"value":549},{"type":58,"tag":465,"props":643,"children":644},{"style":487},[645],{"type":64,"value":554},{"type":58,"tag":465,"props":647,"children":649},{"class":467,"line":648},6,[650],{"type":58,"tag":465,"props":651,"children":653},{"emptyLinePlaceholder":652},true,[654],{"type":64,"value":655},"\n",{"type":58,"tag":465,"props":657,"children":659},{"class":467,"line":658},7,[660,664,668,672,676,680,685,689],{"type":58,"tag":465,"props":661,"children":662},{"style":482},[663],{"type":64,"value":47},{"type":58,"tag":465,"props":665,"children":666},{"style":487},[667],{"type":64,"value":589},{"type":58,"tag":465,"props":669,"children":670},{"style":487},[671],{"type":64,"value":594},{"type":58,"tag":465,"props":673,"children":674},{"style":487},[675],{"type":64,"value":599},{"type":58,"tag":465,"props":677,"children":678},{"style":493},[679],{"type":64,"value":604},{"type":58,"tag":465,"props":681,"children":682},{"style":487},[683],{"type":64,"value":684}," --ranks-per-node",{"type":58,"tag":465,"props":686,"children":687},{"style":493},[688],{"type":64,"value":496},{"type":58,"tag":465,"props":690,"children":691},{"style":519},[692],{"type":64,"value":522},{"type":58,"tag":465,"props":694,"children":696},{"class":467,"line":695},8,[697,702,706,710,714,718,722],{"type":58,"tag":465,"props":698,"children":699},{"style":487},[700],{"type":64,"value":701}," --gpus",{"type":58,"tag":465,"props":703,"children":704},{"style":493},[705],{"type":64,"value":496},{"type":58,"tag":465,"props":707,"children":708},{"style":487},[709],{"type":64,"value":501},{"type":58,"tag":465,"props":711,"children":712},{"style":493},[713],{"type":64,"value":506},{"type":58,"tag":465,"props":715,"children":716},{"style":487},[717],{"type":64,"value":511},{"type":58,"tag":465,"props":719,"children":720},{"style":493},[721],{"type":64,"value":516},{"type":58,"tag":465,"props":723,"children":724},{"style":519},[725],{"type":64,"value":522},{"type":58,"tag":465,"props":727,"children":729},{"class":467,"line":728},9,[730,734],{"type":58,"tag":465,"props":731,"children":732},{"style":487},[733],{"type":64,"value":531},{"type":58,"tag":465,"props":735,"children":736},{"style":519},[737],{"type":64,"value":522},{"type":58,"tag":465,"props":739,"children":741},{"class":467,"line":740},10,[742,746,750],{"type":58,"tag":465,"props":743,"children":744},{"style":487},[745],{"type":64,"value":544},{"type":58,"tag":465,"props":747,"children":748},{"style":487},[749],{"type":64,"value":549},{"type":58,"tag":465,"props":751,"children":752},{"style":487},[753],{"type":64,"value":554},{"type":58,"tag":67,"props":755,"children":756},{},[757,759,764,766,772],{"type":64,"value":758},"No layout flags — the read driver walks every ",{"type":58,"tag":79,"props":760,"children":762},{"className":761},[],[763],{"type":64,"value":92},{"type":64,"value":765}," header to recover\nper-file row counts, the trailing shape, and the dtype, then derives\n",{"type":58,"tag":79,"props":767,"children":769},{"className":768},[],[770],{"type":64,"value":771},"tile_rows",{"type":64,"value":773}," from the available processor count.",{"type":58,"tag":67,"props":775,"children":776},{},[777,783,785,791,793,799,801,806],{"type":58,"tag":79,"props":778,"children":780},{"className":779},[],[781],{"type":64,"value":782},"--min-gpu-chunk 1",{"type":64,"value":784}," is only needed when the per-tile element count is\nbelow Legate's default minimum chunk size for GPU launches (e.g. the\nworked example's defaults — total rows split across 4 GPUs at\n",{"type":58,"tag":79,"props":786,"children":788},{"className":787},[],[789],{"type":64,"value":790},"~1M",{"type":64,"value":792}," per tile — fall below the threshold and would otherwise be\nfolded onto a single GPU). For production-sized datasets (tens of\nmillions of elements per tile or larger) you can drop the flag and\nlet Legate use its default. Bumping it to a moderate value (e.g.\n",{"type":58,"tag":79,"props":794,"children":796},{"className":795},[],[797],{"type":64,"value":798},"--min-gpu-chunk 1024",{"type":64,"value":800},") is fine when each tile is large enough that\nper-task overhead matters more than getting ",{"type":58,"tag":96,"props":802,"children":803},{},[804],{"type":64,"value":805},"every",{"type":64,"value":807}," GPU a tile.",{"type":58,"tag":267,"props":809,"children":811},{"id":810},"instructions",[812],{"type":64,"value":813},"Instructions",{"type":58,"tag":67,"props":815,"children":816},{},[817,819,824],{"type":64,"value":818},"Five steps from a ",{"type":58,"tag":79,"props":820,"children":822},{"className":821},[],[823],{"type":64,"value":92},{"type":64,"value":825}," worked example; only step 1 (parsing the\nformat header) and step 4 (the per-file reader inside the task body)\nare format-specific. The other three (allocate destination, partition,\nfence) are reused unchanged across formats — see \"Other formats\" below\nfor the swap-points.",{"type":58,"tag":827,"props":828,"children":830},"h3",{"id":829},"_1-read-the-metadata-from-every-shard",[831],{"type":64,"value":832},"1. Read the metadata from every shard",{"type":58,"tag":67,"props":834,"children":835},{},[836,838,843,845,851],{"type":64,"value":837},"Scan the directory and peek at every ",{"type":58,"tag":79,"props":839,"children":841},{"className":840},[],[842],{"type":64,"value":92},{"type":64,"value":844}," header (",{"type":58,"tag":79,"props":846,"children":848},{"className":847},[],[849],{"type":64,"value":850},"mmap_mode=\"r\"",{"type":64,"value":852},"\nreads only the header). The header carries the per-shard shape and\ndtype, so the driver can recover total rows, trailing shape, and a\ncumulative row-offset table without ever loading the data:",{"type":58,"tag":454,"props":854,"children":858},{"className":855,"code":856,"language":857,"meta":459,"style":459},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","paths = sorted(SHARD_DIR.glob(\"shard_*.npy\"))\n\nper_file_rows = [] # rows along axis 0 per file\ntrailing_shape = None # shape[1:], must match across files\ndtype = None\nfor p in paths:\n hdr = np.load(p, mmap_mode=\"r\")\n if trailing_shape is None:\n trailing_shape = tuple(hdr.shape[1:])\n dtype = hdr.dtype\n elif tuple(hdr.shape[1:]) != trailing_shape or hdr.dtype != dtype:\n raise RuntimeError(\n f\"{p.name}: trailing shape \u002F dtype mismatch \"\n f\"({hdr.shape[1:]}\u002F{hdr.dtype} vs {trailing_shape}\u002F{dtype})\"\n )\n per_file_rows.append(int(hdr.shape[0]))\n\ncum_rows = np.cumsum([0] + per_file_rows, dtype=np.int64) # length N+1\ntotal_rows = int(cum_rows[-1])\n","python",[859],{"type":58,"tag":79,"props":860,"children":861},{"__ignoreMap":459},[862,870,877,885,893,901,909,917,925,933,941,950,959,968,977,986,995,1003,1012],{"type":58,"tag":465,"props":863,"children":864},{"class":467,"line":468},[865],{"type":58,"tag":465,"props":866,"children":867},{},[868],{"type":64,"value":869},"paths = sorted(SHARD_DIR.glob(\"shard_*.npy\"))\n",{"type":58,"tag":465,"props":871,"children":872},{"class":467,"line":478},[873],{"type":58,"tag":465,"props":874,"children":875},{"emptyLinePlaceholder":652},[876],{"type":64,"value":655},{"type":58,"tag":465,"props":878,"children":879},{"class":467,"line":525},[880],{"type":58,"tag":465,"props":881,"children":882},{},[883],{"type":64,"value":884},"per_file_rows = [] # rows along axis 0 per file\n",{"type":58,"tag":465,"props":886,"children":887},{"class":467,"line":538},[888],{"type":58,"tag":465,"props":889,"children":890},{},[891],{"type":64,"value":892},"trailing_shape = None # shape[1:], must match across files\n",{"type":58,"tag":465,"props":894,"children":895},{"class":467,"line":631},[896],{"type":58,"tag":465,"props":897,"children":898},{},[899],{"type":64,"value":900},"dtype = None\n",{"type":58,"tag":465,"props":902,"children":903},{"class":467,"line":648},[904],{"type":58,"tag":465,"props":905,"children":906},{},[907],{"type":64,"value":908},"for p in paths:\n",{"type":58,"tag":465,"props":910,"children":911},{"class":467,"line":658},[912],{"type":58,"tag":465,"props":913,"children":914},{},[915],{"type":64,"value":916}," hdr = np.load(p, mmap_mode=\"r\")\n",{"type":58,"tag":465,"props":918,"children":919},{"class":467,"line":695},[920],{"type":58,"tag":465,"props":921,"children":922},{},[923],{"type":64,"value":924}," if trailing_shape is None:\n",{"type":58,"tag":465,"props":926,"children":927},{"class":467,"line":728},[928],{"type":58,"tag":465,"props":929,"children":930},{},[931],{"type":64,"value":932}," trailing_shape = tuple(hdr.shape[1:])\n",{"type":58,"tag":465,"props":934,"children":935},{"class":467,"line":740},[936],{"type":58,"tag":465,"props":937,"children":938},{},[939],{"type":64,"value":940}," dtype = hdr.dtype\n",{"type":58,"tag":465,"props":942,"children":944},{"class":467,"line":943},11,[945],{"type":58,"tag":465,"props":946,"children":947},{},[948],{"type":64,"value":949}," elif tuple(hdr.shape[1:]) != trailing_shape or hdr.dtype != dtype:\n",{"type":58,"tag":465,"props":951,"children":953},{"class":467,"line":952},12,[954],{"type":58,"tag":465,"props":955,"children":956},{},[957],{"type":64,"value":958}," raise RuntimeError(\n",{"type":58,"tag":465,"props":960,"children":962},{"class":467,"line":961},13,[963],{"type":58,"tag":465,"props":964,"children":965},{},[966],{"type":64,"value":967}," f\"{p.name}: trailing shape \u002F dtype mismatch \"\n",{"type":58,"tag":465,"props":969,"children":971},{"class":467,"line":970},14,[972],{"type":58,"tag":465,"props":973,"children":974},{},[975],{"type":64,"value":976}," f\"({hdr.shape[1:]}\u002F{hdr.dtype} vs {trailing_shape}\u002F{dtype})\"\n",{"type":58,"tag":465,"props":978,"children":980},{"class":467,"line":979},15,[981],{"type":58,"tag":465,"props":982,"children":983},{},[984],{"type":64,"value":985}," )\n",{"type":58,"tag":465,"props":987,"children":989},{"class":467,"line":988},16,[990],{"type":58,"tag":465,"props":991,"children":992},{},[993],{"type":64,"value":994}," per_file_rows.append(int(hdr.shape[0]))\n",{"type":58,"tag":465,"props":996,"children":998},{"class":467,"line":997},17,[999],{"type":58,"tag":465,"props":1000,"children":1001},{"emptyLinePlaceholder":652},[1002],{"type":64,"value":655},{"type":58,"tag":465,"props":1004,"children":1006},{"class":467,"line":1005},18,[1007],{"type":58,"tag":465,"props":1008,"children":1009},{},[1010],{"type":64,"value":1011},"cum_rows = np.cumsum([0] + per_file_rows, dtype=np.int64) # length N+1\n",{"type":58,"tag":465,"props":1013,"children":1015},{"class":467,"line":1014},19,[1016],{"type":58,"tag":465,"props":1017,"children":1018},{},[1019],{"type":64,"value":1020},"total_rows = int(cum_rows[-1])\n",{"type":58,"tag":67,"props":1022,"children":1023},{},[1024,1026,1031,1033,1039,1041,1046,1048,1053,1055,1061,1063,1068,1070,1075,1077,1083,1085,1091],{"type":64,"value":1025},"The snippet above enforces matching ",{"type":58,"tag":79,"props":1027,"children":1029},{"className":1028},[],[1030],{"type":64,"value":241},{"type":64,"value":1032}," and ",{"type":58,"tag":79,"props":1034,"children":1036},{"className":1035},[],[1037],{"type":64,"value":1038},"trailing_shape",{"type":64,"value":1040}," (i.e.\n",{"type":58,"tag":79,"props":1042,"children":1044},{"className":1043},[],[1045],{"type":64,"value":339},{"type":64,"value":1047},") across files. ",{"type":58,"tag":71,"props":1049,"children":1050},{},[1051],{"type":64,"value":1052},"Per-shard row counts may differ",{"type":64,"value":1054}," — the\ncum-rows table handles that. Production code should also verify that\nnames form a contiguous ",{"type":58,"tag":79,"props":1056,"children":1058},{"className":1057},[],[1059],{"type":64,"value":1060},"shard_0000.npy ... shard_NNNN.npy",{"type":64,"value":1062}," sequence\n(omitted from the snippet for brevity; see ",{"type":58,"tag":79,"props":1064,"children":1066},{"className":1065},[],[1067],{"type":64,"value":364},{"type":64,"value":1069}," in the\nworked example). Discovery relies only on what the\non-disk format itself exposes (the ",{"type":58,"tag":79,"props":1071,"children":1073},{"className":1072},[],[1074],{"type":64,"value":92},{"type":64,"value":1076}," header here, ",{"type":58,"tag":79,"props":1078,"children":1080},{"className":1079},[],[1081],{"type":64,"value":1082},".shape",{"type":64,"value":1084}," \u002F\n",{"type":58,"tag":79,"props":1086,"children":1088},{"className":1087},[],[1089],{"type":64,"value":1090},".dtype",{"type":64,"value":1092}," for HDF5, etc.); any sidecar (manifest, content hashes) is a\nseparate verification step on top.",{"type":58,"tag":827,"props":1094,"children":1096},{"id":1095},"_2-create-the-cupynumeric-output-store-from-the-metadata",[1097],{"type":64,"value":1098},"2. Create the cupynumeric output store from the metadata",{"type":58,"tag":67,"props":1100,"children":1101},{},[1102,1104,1110,1112,1117,1119,1125],{"type":64,"value":1103},"The total array spans ",{"type":58,"tag":79,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":64,"value":1109},"total_rows",{"type":64,"value":1111}," along axis 0; trailing axes come\nfrom ",{"type":58,"tag":79,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":64,"value":1038},{"type":64,"value":1118}," unchanged. Use ",{"type":58,"tag":79,"props":1120,"children":1122},{"className":1121},[],[1123],{"type":64,"value":1124},"cn.empty",{"type":64,"value":1126}," — the task overwrites\nevery cell, zero-init would be wasted.",{"type":58,"tag":454,"props":1128,"children":1130},{"className":855,"code":1129,"language":857,"meta":459,"style":459},"import cupynumeric as cn\n\ntotal_shape = (total_rows,) + trailing_shape\nout = cn.empty(total_shape, dtype=dtype)\n",[1131],{"type":58,"tag":79,"props":1132,"children":1133},{"__ignoreMap":459},[1134,1142,1149,1157],{"type":58,"tag":465,"props":1135,"children":1136},{"class":467,"line":468},[1137],{"type":58,"tag":465,"props":1138,"children":1139},{},[1140],{"type":64,"value":1141},"import cupynumeric as cn\n",{"type":58,"tag":465,"props":1143,"children":1144},{"class":467,"line":478},[1145],{"type":58,"tag":465,"props":1146,"children":1147},{"emptyLinePlaceholder":652},[1148],{"type":64,"value":655},{"type":58,"tag":465,"props":1150,"children":1151},{"class":467,"line":525},[1152],{"type":58,"tag":465,"props":1153,"children":1154},{},[1155],{"type":64,"value":1156},"total_shape = (total_rows,) + trailing_shape\n",{"type":58,"tag":465,"props":1158,"children":1159},{"class":467,"line":538},[1160],{"type":58,"tag":465,"props":1161,"children":1162},{},[1163],{"type":64,"value":1164},"out = cn.empty(total_shape, dtype=dtype)\n",{"type":58,"tag":827,"props":1166,"children":1168},{"id":1167},"_3-tile-the-store-by-processor-count",[1169],{"type":64,"value":1170},"3. Tile the store by processor count",{"type":58,"tag":67,"props":1172,"children":1173},{},[1174,1176,1181,1183,1189,1191,1197],{"type":64,"value":1175},"The launch shape is sized to the ",{"type":58,"tag":71,"props":1177,"children":1178},{},[1179],{"type":64,"value":1180},"available processors",{"type":64,"value":1182},", not to the\nfile count. Pick ",{"type":58,"tag":79,"props":1184,"children":1186},{"className":1185},[],[1187],{"type":64,"value":1188},"tile_rows = ceil(total_rows \u002F num_processors)",{"type":64,"value":1190}," and\npartition axis 0 by that tile size. Trailing axes are not partitioned\n(tile spans the full extent there). The last tile is allowed to be\nshort — that's exactly what ",{"type":58,"tag":79,"props":1192,"children":1194},{"className":1193},[],[1195],{"type":64,"value":1196},"partition_by_tiling",{"type":64,"value":1198}," supports — so the\nrecipe needs no divisibility constraint.",{"type":58,"tag":454,"props":1200,"children":1202},{"className":855,"code":1201,"language":857,"meta":459,"style":459},"from legate.core import TaskTarget, get_legate_runtime\nfrom legate.core.data_interface import as_logical_array\n\nruntime = get_legate_runtime()\nmachine = runtime.get_machine()\nnum_processors = max(\n machine.count(TaskTarget.GPU),\n machine.count(TaskTarget.OMP),\n machine.count(TaskTarget.CPU),\n 1,\n)\n\ntile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\ntile_shape = (tile_rows,) + trailing_shape\npartition = as_logical_array(out).data.partition_by_tiling(tile_shape)\n\nnum_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows # match partition tile count\n",[1203],{"type":58,"tag":79,"props":1204,"children":1205},{"__ignoreMap":459},[1206,1214,1222,1229,1237,1245,1253,1261,1269,1277,1285,1293,1300,1308,1316,1324,1331],{"type":58,"tag":465,"props":1207,"children":1208},{"class":467,"line":468},[1209],{"type":58,"tag":465,"props":1210,"children":1211},{},[1212],{"type":64,"value":1213},"from legate.core import TaskTarget, get_legate_runtime\n",{"type":58,"tag":465,"props":1215,"children":1216},{"class":467,"line":478},[1217],{"type":58,"tag":465,"props":1218,"children":1219},{},[1220],{"type":64,"value":1221},"from legate.core.data_interface import as_logical_array\n",{"type":58,"tag":465,"props":1223,"children":1224},{"class":467,"line":525},[1225],{"type":58,"tag":465,"props":1226,"children":1227},{"emptyLinePlaceholder":652},[1228],{"type":64,"value":655},{"type":58,"tag":465,"props":1230,"children":1231},{"class":467,"line":538},[1232],{"type":58,"tag":465,"props":1233,"children":1234},{},[1235],{"type":64,"value":1236},"runtime = get_legate_runtime()\n",{"type":58,"tag":465,"props":1238,"children":1239},{"class":467,"line":631},[1240],{"type":58,"tag":465,"props":1241,"children":1242},{},[1243],{"type":64,"value":1244},"machine = runtime.get_machine()\n",{"type":58,"tag":465,"props":1246,"children":1247},{"class":467,"line":648},[1248],{"type":58,"tag":465,"props":1249,"children":1250},{},[1251],{"type":64,"value":1252},"num_processors = max(\n",{"type":58,"tag":465,"props":1254,"children":1255},{"class":467,"line":658},[1256],{"type":58,"tag":465,"props":1257,"children":1258},{},[1259],{"type":64,"value":1260}," machine.count(TaskTarget.GPU),\n",{"type":58,"tag":465,"props":1262,"children":1263},{"class":467,"line":695},[1264],{"type":58,"tag":465,"props":1265,"children":1266},{},[1267],{"type":64,"value":1268}," machine.count(TaskTarget.OMP),\n",{"type":58,"tag":465,"props":1270,"children":1271},{"class":467,"line":728},[1272],{"type":58,"tag":465,"props":1273,"children":1274},{},[1275],{"type":64,"value":1276}," machine.count(TaskTarget.CPU),\n",{"type":58,"tag":465,"props":1278,"children":1279},{"class":467,"line":740},[1280],{"type":58,"tag":465,"props":1281,"children":1282},{},[1283],{"type":64,"value":1284}," 1,\n",{"type":58,"tag":465,"props":1286,"children":1287},{"class":467,"line":943},[1288],{"type":58,"tag":465,"props":1289,"children":1290},{},[1291],{"type":64,"value":1292},")\n",{"type":58,"tag":465,"props":1294,"children":1295},{"class":467,"line":952},[1296],{"type":58,"tag":465,"props":1297,"children":1298},{"emptyLinePlaceholder":652},[1299],{"type":64,"value":655},{"type":58,"tag":465,"props":1301,"children":1302},{"class":467,"line":961},[1303],{"type":58,"tag":465,"props":1304,"children":1305},{},[1306],{"type":64,"value":1307},"tile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\n",{"type":58,"tag":465,"props":1309,"children":1310},{"class":467,"line":970},[1311],{"type":58,"tag":465,"props":1312,"children":1313},{},[1314],{"type":64,"value":1315},"tile_shape = (tile_rows,) + trailing_shape\n",{"type":58,"tag":465,"props":1317,"children":1318},{"class":467,"line":979},[1319],{"type":58,"tag":465,"props":1320,"children":1321},{},[1322],{"type":64,"value":1323},"partition = as_logical_array(out).data.partition_by_tiling(tile_shape)\n",{"type":58,"tag":465,"props":1325,"children":1326},{"class":467,"line":988},[1327],{"type":58,"tag":465,"props":1328,"children":1329},{"emptyLinePlaceholder":652},[1330],{"type":64,"value":655},{"type":58,"tag":465,"props":1332,"children":1333},{"class":467,"line":997},[1334],{"type":58,"tag":465,"props":1335,"children":1336},{},[1337],{"type":64,"value":1338},"num_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows # match partition tile count\n",{"type":58,"tag":827,"props":1340,"children":1342},{"id":1341},"_4-define-the-leaf-task-and-launch-it-manually",[1343],{"type":64,"value":1344},"4. Define the leaf task and launch it manually",{"type":58,"tag":67,"props":1346,"children":1347},{},[1348,1354,1355,1361,1363,1369],{"type":58,"tag":79,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":64,"value":1353},"PATHS",{"type":64,"value":1032},{"type":58,"tag":79,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":64,"value":1360},"CUM_ROWS",{"type":64,"value":1362}," (the file paths and cumulative row-offset\ntable from step 1) plus ",{"type":58,"tag":79,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":64,"value":1368},"TILE_ROWS",{"type":64,"value":1370}," are populated as module globals\nby the driver before launching; control replication runs the driver\non every rank, so every worker sees identical values.",{"type":58,"tag":67,"props":1372,"children":1373},{},[1374,1376,1382,1384,1390,1392,1397,1399,1405,1407,1413,1415,1421,1423,1429,1431,1437],{"type":64,"value":1375},"Each task builds its consumer view first (cupy on GPU, numpy on\nCPU\u002FOMP) and reads the tile's actual row count from ",{"type":58,"tag":79,"props":1377,"children":1379},{"className":1378},[],[1380],{"type":64,"value":1381},"view.shape[0]",{"type":64,"value":1383},"\n— ",{"type":58,"tag":79,"props":1385,"children":1387},{"className":1386},[],[1388],{"type":64,"value":1389},"PhysicalStore",{"type":64,"value":1391}," itself has no ",{"type":58,"tag":79,"props":1393,"children":1395},{"className":1394},[],[1396],{"type":64,"value":1082},{"type":64,"value":1398}," attribute, so going through\nthe view is required. It then computes its global row range from its\nlaunch coordinate and that row count, bisects ",{"type":58,"tag":79,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":64,"value":1404},"cum_rows",{"type":64,"value":1406}," for the\noverlapping file(s), and copies each overlapping file slice into the\nmatching destination slice. Register CPU, OMP, and GPU variants so\nthe same launch runs unchanged anywhere; dispatch on\n",{"type":58,"tag":79,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":64,"value":1412},"ctx.get_variant_kind()",{"type":64,"value":1414}," picks the consumer matching where the\n",{"type":58,"tag":79,"props":1416,"children":1418},{"className":1417},[],[1419],{"type":64,"value":1420},"OutputStore",{"type":64,"value":1422}," is resident (",{"type":58,"tag":79,"props":1424,"children":1426},{"className":1425},[],[1427],{"type":64,"value":1428},"cp.from_dlpack(dst)",{"type":64,"value":1430}," for FBMEM,\n",{"type":58,"tag":79,"props":1432,"children":1434},{"className":1433},[],[1435],{"type":64,"value":1436},"np.asarray(dst)",{"type":64,"value":1438}," for SYSMEM). cupy is imported inside the GPU\nbranch only, so the task body loads on machines without cupy.",{"type":58,"tag":454,"props":1440,"children":1442},{"className":855,"code":1441,"language":857,"meta":459,"style":459},"import bisect\nfrom legate.core import TaskContext, VariantCode\nfrom legate.core.task import OutputStore, task\n\n@task(variants=(VariantCode.CPU, VariantCode.OMP, VariantCode.GPU))\ndef load_tile(ctx: TaskContext, dst: OutputStore) -> None:\n t = ctx.task_index[0] # tile index 0..num_tasks-1\n\n variant = ctx.get_variant_kind()\n if variant == VariantCode.GPU:\n import cupy as cp # lazy: only on GPU\n view = cp.from_dlpack(dst)\n else:\n view = np.asarray(dst) # zero-copy numpy view\n\n tile_rows_actual = view.shape[0] # short on the last tile\n row_start = t * TILE_ROWS # global axis-0 start\n row_end = row_start + tile_rows_actual\n\n # Find the half-open range of file indices that overlap [row_start, row_end).\n first_file = bisect.bisect_right(CUM_ROWS, row_start) - 1\n last_file = bisect.bisect_right(CUM_ROWS, row_end - 1) - 1\n\n for f in range(first_file, last_file + 1):\n # Intersection of tile [row_start, row_end) with file [cum[f], cum[f+1]).\n lo = max(row_start, int(CUM_ROWS[f]))\n hi = min(row_end, int(CUM_ROWS[f + 1]))\n file_lo = lo - int(CUM_ROWS[f])\n file_hi = hi - int(CUM_ROWS[f])\n dst_lo = lo - row_start\n dst_hi = hi - row_start\n chunk = np.ascontiguousarray(\n np.load(PATHS[f], mmap_mode=\"r\")[file_lo:file_hi]\n )\n if variant == VariantCode.GPU:\n view[dst_lo:dst_hi].set(chunk) # cudaMemcpyAsync H2D\n else:\n view[dst_lo:dst_hi] = chunk # zero-copy numpy write\n\nmanual_task = runtime.create_manual_task(\n load_tile.library,\n load_tile.task_id,\n (num_tasks,), # launch domain == tile count\n)\nmanual_task.add_output(partition)\nmanual_task.execute()\n",[1443],{"type":58,"tag":79,"props":1444,"children":1445},{"__ignoreMap":459},[1446,1454,1462,1470,1477,1485,1493,1501,1508,1516,1524,1532,1540,1548,1556,1563,1571,1579,1587,1594,1603,1612,1621,1629,1638,1647,1656,1665,1674,1683,1692,1701,1710,1719,1727,1736,1745,1754,1763,1771,1780,1789,1798,1807,1815,1824],{"type":58,"tag":465,"props":1447,"children":1448},{"class":467,"line":468},[1449],{"type":58,"tag":465,"props":1450,"children":1451},{},[1452],{"type":64,"value":1453},"import bisect\n",{"type":58,"tag":465,"props":1455,"children":1456},{"class":467,"line":478},[1457],{"type":58,"tag":465,"props":1458,"children":1459},{},[1460],{"type":64,"value":1461},"from legate.core import TaskContext, VariantCode\n",{"type":58,"tag":465,"props":1463,"children":1464},{"class":467,"line":525},[1465],{"type":58,"tag":465,"props":1466,"children":1467},{},[1468],{"type":64,"value":1469},"from legate.core.task import OutputStore, task\n",{"type":58,"tag":465,"props":1471,"children":1472},{"class":467,"line":538},[1473],{"type":58,"tag":465,"props":1474,"children":1475},{"emptyLinePlaceholder":652},[1476],{"type":64,"value":655},{"type":58,"tag":465,"props":1478,"children":1479},{"class":467,"line":631},[1480],{"type":58,"tag":465,"props":1481,"children":1482},{},[1483],{"type":64,"value":1484},"@task(variants=(VariantCode.CPU, VariantCode.OMP, VariantCode.GPU))\n",{"type":58,"tag":465,"props":1486,"children":1487},{"class":467,"line":648},[1488],{"type":58,"tag":465,"props":1489,"children":1490},{},[1491],{"type":64,"value":1492},"def load_tile(ctx: TaskContext, dst: OutputStore) -> None:\n",{"type":58,"tag":465,"props":1494,"children":1495},{"class":467,"line":658},[1496],{"type":58,"tag":465,"props":1497,"children":1498},{},[1499],{"type":64,"value":1500}," t = ctx.task_index[0] # tile index 0..num_tasks-1\n",{"type":58,"tag":465,"props":1502,"children":1503},{"class":467,"line":695},[1504],{"type":58,"tag":465,"props":1505,"children":1506},{"emptyLinePlaceholder":652},[1507],{"type":64,"value":655},{"type":58,"tag":465,"props":1509,"children":1510},{"class":467,"line":728},[1511],{"type":58,"tag":465,"props":1512,"children":1513},{},[1514],{"type":64,"value":1515}," variant = ctx.get_variant_kind()\n",{"type":58,"tag":465,"props":1517,"children":1518},{"class":467,"line":740},[1519],{"type":58,"tag":465,"props":1520,"children":1521},{},[1522],{"type":64,"value":1523}," if variant == VariantCode.GPU:\n",{"type":58,"tag":465,"props":1525,"children":1526},{"class":467,"line":943},[1527],{"type":58,"tag":465,"props":1528,"children":1529},{},[1530],{"type":64,"value":1531}," import cupy as cp # lazy: only on GPU\n",{"type":58,"tag":465,"props":1533,"children":1534},{"class":467,"line":952},[1535],{"type":58,"tag":465,"props":1536,"children":1537},{},[1538],{"type":64,"value":1539}," view = cp.from_dlpack(dst)\n",{"type":58,"tag":465,"props":1541,"children":1542},{"class":467,"line":961},[1543],{"type":58,"tag":465,"props":1544,"children":1545},{},[1546],{"type":64,"value":1547}," else:\n",{"type":58,"tag":465,"props":1549,"children":1550},{"class":467,"line":970},[1551],{"type":58,"tag":465,"props":1552,"children":1553},{},[1554],{"type":64,"value":1555}," view = np.asarray(dst) # zero-copy numpy view\n",{"type":58,"tag":465,"props":1557,"children":1558},{"class":467,"line":979},[1559],{"type":58,"tag":465,"props":1560,"children":1561},{"emptyLinePlaceholder":652},[1562],{"type":64,"value":655},{"type":58,"tag":465,"props":1564,"children":1565},{"class":467,"line":988},[1566],{"type":58,"tag":465,"props":1567,"children":1568},{},[1569],{"type":64,"value":1570}," tile_rows_actual = view.shape[0] # short on the last tile\n",{"type":58,"tag":465,"props":1572,"children":1573},{"class":467,"line":997},[1574],{"type":58,"tag":465,"props":1575,"children":1576},{},[1577],{"type":64,"value":1578}," row_start = t * TILE_ROWS # global axis-0 start\n",{"type":58,"tag":465,"props":1580,"children":1581},{"class":467,"line":1005},[1582],{"type":58,"tag":465,"props":1583,"children":1584},{},[1585],{"type":64,"value":1586}," row_end = row_start + tile_rows_actual\n",{"type":58,"tag":465,"props":1588,"children":1589},{"class":467,"line":1014},[1590],{"type":58,"tag":465,"props":1591,"children":1592},{"emptyLinePlaceholder":652},[1593],{"type":64,"value":655},{"type":58,"tag":465,"props":1595,"children":1597},{"class":467,"line":1596},20,[1598],{"type":58,"tag":465,"props":1599,"children":1600},{},[1601],{"type":64,"value":1602}," # Find the half-open range of file indices that overlap [row_start, row_end).\n",{"type":58,"tag":465,"props":1604,"children":1606},{"class":467,"line":1605},21,[1607],{"type":58,"tag":465,"props":1608,"children":1609},{},[1610],{"type":64,"value":1611}," first_file = bisect.bisect_right(CUM_ROWS, row_start) - 1\n",{"type":58,"tag":465,"props":1613,"children":1615},{"class":467,"line":1614},22,[1616],{"type":58,"tag":465,"props":1617,"children":1618},{},[1619],{"type":64,"value":1620}," last_file = bisect.bisect_right(CUM_ROWS, row_end - 1) - 1\n",{"type":58,"tag":465,"props":1622,"children":1624},{"class":467,"line":1623},23,[1625],{"type":58,"tag":465,"props":1626,"children":1627},{"emptyLinePlaceholder":652},[1628],{"type":64,"value":655},{"type":58,"tag":465,"props":1630,"children":1632},{"class":467,"line":1631},24,[1633],{"type":58,"tag":465,"props":1634,"children":1635},{},[1636],{"type":64,"value":1637}," for f in range(first_file, last_file + 1):\n",{"type":58,"tag":465,"props":1639,"children":1641},{"class":467,"line":1640},25,[1642],{"type":58,"tag":465,"props":1643,"children":1644},{},[1645],{"type":64,"value":1646}," # Intersection of tile [row_start, row_end) with file [cum[f], cum[f+1]).\n",{"type":58,"tag":465,"props":1648,"children":1650},{"class":467,"line":1649},26,[1651],{"type":58,"tag":465,"props":1652,"children":1653},{},[1654],{"type":64,"value":1655}," lo = max(row_start, int(CUM_ROWS[f]))\n",{"type":58,"tag":465,"props":1657,"children":1659},{"class":467,"line":1658},27,[1660],{"type":58,"tag":465,"props":1661,"children":1662},{},[1663],{"type":64,"value":1664}," hi = min(row_end, int(CUM_ROWS[f + 1]))\n",{"type":58,"tag":465,"props":1666,"children":1668},{"class":467,"line":1667},28,[1669],{"type":58,"tag":465,"props":1670,"children":1671},{},[1672],{"type":64,"value":1673}," file_lo = lo - int(CUM_ROWS[f])\n",{"type":58,"tag":465,"props":1675,"children":1677},{"class":467,"line":1676},29,[1678],{"type":58,"tag":465,"props":1679,"children":1680},{},[1681],{"type":64,"value":1682}," file_hi = hi - int(CUM_ROWS[f])\n",{"type":58,"tag":465,"props":1684,"children":1686},{"class":467,"line":1685},30,[1687],{"type":58,"tag":465,"props":1688,"children":1689},{},[1690],{"type":64,"value":1691}," dst_lo = lo - row_start\n",{"type":58,"tag":465,"props":1693,"children":1695},{"class":467,"line":1694},31,[1696],{"type":58,"tag":465,"props":1697,"children":1698},{},[1699],{"type":64,"value":1700}," dst_hi = hi - row_start\n",{"type":58,"tag":465,"props":1702,"children":1704},{"class":467,"line":1703},32,[1705],{"type":58,"tag":465,"props":1706,"children":1707},{},[1708],{"type":64,"value":1709}," chunk = np.ascontiguousarray(\n",{"type":58,"tag":465,"props":1711,"children":1713},{"class":467,"line":1712},33,[1714],{"type":58,"tag":465,"props":1715,"children":1716},{},[1717],{"type":64,"value":1718}," np.load(PATHS[f], mmap_mode=\"r\")[file_lo:file_hi]\n",{"type":58,"tag":465,"props":1720,"children":1722},{"class":467,"line":1721},34,[1723],{"type":58,"tag":465,"props":1724,"children":1725},{},[1726],{"type":64,"value":985},{"type":58,"tag":465,"props":1728,"children":1730},{"class":467,"line":1729},35,[1731],{"type":58,"tag":465,"props":1732,"children":1733},{},[1734],{"type":64,"value":1735}," if variant == VariantCode.GPU:\n",{"type":58,"tag":465,"props":1737,"children":1739},{"class":467,"line":1738},36,[1740],{"type":58,"tag":465,"props":1741,"children":1742},{},[1743],{"type":64,"value":1744}," view[dst_lo:dst_hi].set(chunk) # cudaMemcpyAsync H2D\n",{"type":58,"tag":465,"props":1746,"children":1748},{"class":467,"line":1747},37,[1749],{"type":58,"tag":465,"props":1750,"children":1751},{},[1752],{"type":64,"value":1753}," else:\n",{"type":58,"tag":465,"props":1755,"children":1757},{"class":467,"line":1756},38,[1758],{"type":58,"tag":465,"props":1759,"children":1760},{},[1761],{"type":64,"value":1762}," view[dst_lo:dst_hi] = chunk # zero-copy numpy write\n",{"type":58,"tag":465,"props":1764,"children":1766},{"class":467,"line":1765},39,[1767],{"type":58,"tag":465,"props":1768,"children":1769},{"emptyLinePlaceholder":652},[1770],{"type":64,"value":655},{"type":58,"tag":465,"props":1772,"children":1774},{"class":467,"line":1773},40,[1775],{"type":58,"tag":465,"props":1776,"children":1777},{},[1778],{"type":64,"value":1779},"manual_task = runtime.create_manual_task(\n",{"type":58,"tag":465,"props":1781,"children":1783},{"class":467,"line":1782},41,[1784],{"type":58,"tag":465,"props":1785,"children":1786},{},[1787],{"type":64,"value":1788}," load_tile.library,\n",{"type":58,"tag":465,"props":1790,"children":1792},{"class":467,"line":1791},42,[1793],{"type":58,"tag":465,"props":1794,"children":1795},{},[1796],{"type":64,"value":1797}," load_tile.task_id,\n",{"type":58,"tag":465,"props":1799,"children":1801},{"class":467,"line":1800},43,[1802],{"type":58,"tag":465,"props":1803,"children":1804},{},[1805],{"type":64,"value":1806}," (num_tasks,), # launch domain == tile count\n",{"type":58,"tag":465,"props":1808,"children":1810},{"class":467,"line":1809},44,[1811],{"type":58,"tag":465,"props":1812,"children":1813},{},[1814],{"type":64,"value":1292},{"type":58,"tag":465,"props":1816,"children":1818},{"class":467,"line":1817},45,[1819],{"type":58,"tag":465,"props":1820,"children":1821},{},[1822],{"type":64,"value":1823},"manual_task.add_output(partition)\n",{"type":58,"tag":465,"props":1825,"children":1827},{"class":467,"line":1826},46,[1828],{"type":58,"tag":465,"props":1829,"children":1830},{},[1831],{"type":64,"value":1832},"manual_task.execute()\n",{"type":58,"tag":67,"props":1834,"children":1835},{},[1836,1838,1843,1845,1851,1853,1859,1861,1867,1869,1875],{"type":64,"value":1837},"Both consumers go through ",{"type":58,"tag":79,"props":1839,"children":1841},{"className":1840},[],[1842],{"type":64,"value":1389},{"type":64,"value":1844},"'s native producers\n(",{"type":58,"tag":79,"props":1846,"children":1848},{"className":1847},[],[1849],{"type":64,"value":1850},"__dlpack__",{"type":64,"value":1852}," for cupy, ",{"type":58,"tag":79,"props":1854,"children":1856},{"className":1855},[],[1857],{"type":64,"value":1858},"__array_interface__",{"type":64,"value":1860}," for ",{"type":58,"tag":79,"props":1862,"children":1864},{"className":1863},[],[1865],{"type":64,"value":1866},"np.asarray",{"type":64,"value":1868},") —\nzero-copy views of the local tile. Bisect cost is ",{"type":58,"tag":79,"props":1870,"children":1872},{"className":1871},[],[1873],{"type":64,"value":1874},"O(log num_shards)",{"type":64,"value":1876},"\nand the inner loop typically iterates 1–2 times (tiles overlap at\nmost a couple of files).",{"type":58,"tag":827,"props":1878,"children":1880},{"id":1879},"_5-fence-and-verify",[1881],{"type":64,"value":1882},"5. Fence and verify",{"type":58,"tag":454,"props":1884,"children":1886},{"className":855,"code":1885,"language":857,"meta":459,"style":459},"get_legate_runtime().issue_execution_fence(block=True)\n",[1887],{"type":58,"tag":79,"props":1888,"children":1889},{"__ignoreMap":459},[1890],{"type":58,"tag":465,"props":1891,"children":1892},{"class":467,"line":468},[1893],{"type":58,"tag":465,"props":1894,"children":1895},{},[1896],{"type":64,"value":1885},{"type":58,"tag":267,"props":1898,"children":1900},{"id":1899},"hard-constraints",[1901],{"type":64,"value":1902},"Hard constraints",{"type":58,"tag":1904,"props":1905,"children":1906},"ol",{},[1907,1953,1985,2011],{"type":58,"tag":302,"props":1908,"children":1909},{},[1910,1929,1931,1936,1938,1944,1946,1951],{"type":58,"tag":71,"props":1911,"children":1912},{},[1913,1915,1920,1922,1927],{"type":64,"value":1914},"All shards must share ",{"type":58,"tag":79,"props":1916,"children":1918},{"className":1917},[],[1919],{"type":64,"value":241},{"type":64,"value":1921}," and trailing axes (",{"type":58,"tag":79,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":64,"value":339},{"type":64,"value":1928},").",{"type":64,"value":1930},"\nThe recipe stacks shards along axis 0; the destination's trailing\naxes come from ",{"type":58,"tag":79,"props":1932,"children":1934},{"className":1933},[],[1935],{"type":64,"value":1038},{"type":64,"value":1937},", which the discovery step locks to\nthe value of the first file. Per-shard row counts (",{"type":58,"tag":79,"props":1939,"children":1941},{"className":1940},[],[1942],{"type":64,"value":1943},"shape[0]",{"type":64,"value":1945},") may\nfreely differ — the cumulative-offset table handles them. The\nexample rejects any shard whose ",{"type":58,"tag":79,"props":1947,"children":1949},{"className":1948},[],[1950],{"type":64,"value":241},{"type":64,"value":1952}," or trailing shape differs\nfrom the first one with a descriptive error.",{"type":58,"tag":302,"props":1954,"children":1955},{},[1956,1961,1963,1969,1971,1976,1978,1983],{"type":58,"tag":71,"props":1957,"children":1958},{},[1959],{"type":64,"value":1960},"Pick the consumer that matches the variant.",{"type":64,"value":1962}," ",{"type":58,"tag":79,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":64,"value":1968},"cp.from_dlpack",{"type":64,"value":1970},"\nrejects SYSMEM-resident stores; ",{"type":58,"tag":79,"props":1972,"children":1974},{"className":1973},[],[1975],{"type":64,"value":1866},{"type":64,"value":1977}," silently returns a\nhost view of an FBMEM-resident store you can't actually write\nthrough. Dispatch on ",{"type":58,"tag":79,"props":1979,"children":1981},{"className":1980},[],[1982],{"type":64,"value":1412},{"type":64,"value":1984}," so each variant uses\nits own consumer — see step 4.",{"type":58,"tag":302,"props":1986,"children":1987},{},[1988,1993,1995,2001,2003,2009],{"type":58,"tag":71,"props":1989,"children":1990},{},[1991],{"type":64,"value":1992},"mmap views aren't always C-contiguous",{"type":64,"value":1994}," — wrap each per-file\nslice with ",{"type":58,"tag":79,"props":1996,"children":1998},{"className":1997},[],[1999],{"type":64,"value":2000},"np.ascontiguousarray(arr[file_lo:file_hi])",{"type":64,"value":2002}," before\n",{"type":58,"tag":79,"props":2004,"children":2006},{"className":2005},[],[2007],{"type":64,"value":2008},".set()",{"type":64,"value":2010}," or the numpy in-place write.",{"type":58,"tag":302,"props":2012,"children":2013},{},[2014,2027,2029,2035],{"type":58,"tag":71,"props":2015,"children":2016},{},[2017,2019,2025],{"type":64,"value":2018},"Multi-node: ",{"type":58,"tag":79,"props":2020,"children":2022},{"className":2021},[],[2023],{"type":64,"value":2024},"SHARD_DIR",{"type":64,"value":2026}," must be on a shared filesystem.",{"type":64,"value":2028}," Every\nworker (on every rank) opens shards by path; node-local ",{"type":58,"tag":79,"props":2030,"children":2032},{"className":2031},[],[2033],{"type":64,"value":2034},"\u002Ftmp",{"type":64,"value":2036}," paths\nonly work for single-node demos.",{"type":58,"tag":267,"props":2038,"children":2040},{"id":2039},"variants",[2041],{"type":64,"value":2042},"Variants",{"type":58,"tag":827,"props":2044,"children":2046},{"id":2045},"uniform-shard-fast-path-one-task-per-file",[2047],{"type":64,"value":2048},"Uniform-shard fast path (one task per file)",{"type":58,"tag":67,"props":2050,"children":2051},{},[2052,2054,2060,2062,2068,2070,2076,2078,2084],{"type":64,"value":2053},"When every shard already has the same ",{"type":58,"tag":79,"props":2055,"children":2057},{"className":2056},[],[2058],{"type":64,"value":2059},"(shape, dtype)",{"type":64,"value":2061}," and you happen\nto have ",{"type":58,"tag":79,"props":2063,"children":2065},{"className":2064},[],[2066],{"type":64,"value":2067},"num_shards",{"type":64,"value":2069}," processors available, the cum-rows \u002F bisect\nmachinery is overhead. Set ",{"type":58,"tag":79,"props":2071,"children":2073},{"className":2072},[],[2074],{"type":64,"value":2075},"tile_rows = shard_shape[0]",{"type":64,"value":2077}," and\n",{"type":58,"tag":79,"props":2079,"children":2081},{"className":2080},[],[2082],{"type":64,"value":2083},"num_tasks = num_shards",{"type":64,"value":2085},"; the partition then has one tile per file\nand each task reads exactly one file end-to-end (no bisect, no inner\nloop). The driver-side switch is a one-liner:",{"type":58,"tag":454,"props":2087,"children":2089},{"className":855,"code":2088,"language":857,"meta":459,"style":459},"if all(r == per_file_rows[0] for r in per_file_rows) and num_shards == num_processors:\n tile_rows = per_file_rows[0]\nelse:\n tile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\n",[2090],{"type":58,"tag":79,"props":2091,"children":2092},{"__ignoreMap":459},[2093,2101,2109,2117],{"type":58,"tag":465,"props":2094,"children":2095},{"class":467,"line":468},[2096],{"type":58,"tag":465,"props":2097,"children":2098},{},[2099],{"type":64,"value":2100},"if all(r == per_file_rows[0] for r in per_file_rows) and num_shards == num_processors:\n",{"type":58,"tag":465,"props":2102,"children":2103},{"class":467,"line":478},[2104],{"type":58,"tag":465,"props":2105,"children":2106},{},[2107],{"type":64,"value":2108}," tile_rows = per_file_rows[0]\n",{"type":58,"tag":465,"props":2110,"children":2111},{"class":467,"line":525},[2112],{"type":58,"tag":465,"props":2113,"children":2114},{},[2115],{"type":64,"value":2116},"else:\n",{"type":58,"tag":465,"props":2118,"children":2119},{"class":467,"line":538},[2120],{"type":58,"tag":465,"props":2121,"children":2122},{},[2123],{"type":64,"value":2124}," tile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\n",{"type":58,"tag":67,"props":2126,"children":2127},{},[2128,2130,2136],{"type":64,"value":2129},"The same ",{"type":58,"tag":79,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":64,"value":2135},"load_tile",{"type":64,"value":2137}," task body still works in either mode — the inner\nloop just happens to iterate exactly once per task. There's no need\nfor a separate task body for the fast path.",{"type":58,"tag":827,"props":2139,"children":2141},{"id":2140},"over-decompose-for-better-load-balancing",[2142],{"type":64,"value":2143},"Over-decompose for better load balancing",{"type":58,"tag":67,"props":2145,"children":2146},{},[2147,2149,2154,2156,2162,2164,2170],{"type":64,"value":2148},"The default ",{"type":58,"tag":79,"props":2150,"children":2152},{"className":2151},[],[2153],{"type":64,"value":1188},{"type":64,"value":2155}," gives one\ntile per processor. To over-decompose by a factor ",{"type":58,"tag":79,"props":2157,"children":2159},{"className":2158},[],[2160],{"type":64,"value":2161},"K",{"type":64,"value":2163}," (smaller tiles,\nmore point tasks, finer-grained queueing), divide by ",{"type":58,"tag":79,"props":2165,"children":2167},{"className":2166},[],[2168],{"type":64,"value":2169},"K * num_processors",{"type":64,"value":2171},"\ninstead:",{"type":58,"tag":454,"props":2173,"children":2175},{"className":855,"code":2174,"language":857,"meta":459,"style":459},"tile_rows = max(1, (total_rows + K * num_processors - 1) \u002F\u002F (K * num_processors))\n",[2176],{"type":58,"tag":79,"props":2177,"children":2178},{"__ignoreMap":459},[2179],{"type":58,"tag":465,"props":2180,"children":2181},{"class":467,"line":468},[2182],{"type":58,"tag":465,"props":2183,"children":2184},{},[2185],{"type":64,"value":2174},{"type":58,"tag":67,"props":2187,"children":2188},{},[2189,2195,2197,2202],{"type":58,"tag":79,"props":2190,"children":2192},{"className":2191},[],[2193],{"type":64,"value":2194},"num_tasks = ceil(total_rows \u002F tile_rows)",{"type":64,"value":2196}," then expands to roughly\n",{"type":58,"tag":79,"props":2198,"children":2200},{"className":2199},[],[2201],{"type":64,"value":2169},{"type":64,"value":2203},". The same task body still works — bisect just lands\non more tasks per file.",{"type":58,"tag":827,"props":2205,"children":2207},{"id":2206},"other-formats",[2208],{"type":64,"value":2209},"Other formats",{"type":58,"tag":67,"props":2211,"children":2212},{},[2213,2215,2220,2222,2228,2230,2236],{"type":64,"value":2214},"Only the per-file reader inside ",{"type":58,"tag":79,"props":2216,"children":2218},{"className":2217},[],[2219],{"type":64,"value":2135},{"type":64,"value":2221}," changes. The reader's\ncontract: given a file path and a half-open row range\n",{"type":58,"tag":79,"props":2223,"children":2225},{"className":2224},[],[2226],{"type":64,"value":2227},"[file_lo, file_hi)",{"type":64,"value":2229}," along axis 0, return a numpy array of shape\n",{"type":58,"tag":79,"props":2231,"children":2233},{"className":2232},[],[2234],{"type":64,"value":2235},"(file_hi - file_lo,) + trailing_shape",{"type":64,"value":2237}," that can be made C-contiguous.\nCheap range\u002Fslice reads are required — formats that only support\n\"read the whole file\" defeat the partial-overlap case (a tile that\ncovers only part of one file).",{"type":58,"tag":104,"props":2239,"children":2240},{},[2241,2256],{"type":58,"tag":108,"props":2242,"children":2243},{},[2244],{"type":58,"tag":112,"props":2245,"children":2246},{},[2247,2251],{"type":58,"tag":116,"props":2248,"children":2249},{},[2250],{"type":64,"value":120},{"type":58,"tag":116,"props":2252,"children":2253},{},[2254],{"type":64,"value":2255},"Reader inside the leaf task",{"type":58,"tag":127,"props":2257,"children":2258},{},[2259,2284,2306,2326],{"type":58,"tag":112,"props":2260,"children":2261},{},[2262,2275],{"type":58,"tag":134,"props":2263,"children":2264},{},[2265,2273],{"type":58,"tag":71,"props":2266,"children":2267},{},[2268],{"type":58,"tag":79,"props":2269,"children":2271},{"className":2270},[],[2272],{"type":64,"value":92},{"type":64,"value":2274}," (worked example)",{"type":58,"tag":134,"props":2276,"children":2277},{},[2278],{"type":58,"tag":79,"props":2279,"children":2281},{"className":2280},[],[2282],{"type":64,"value":2283},"host = np.ascontiguousarray(np.load(p, mmap_mode=\"r\")[file_lo:file_hi])",{"type":58,"tag":112,"props":2285,"children":2286},{},[2287,2297],{"type":58,"tag":134,"props":2288,"children":2289},{},[2290,2295],{"type":58,"tag":71,"props":2291,"children":2292},{},[2293],{"type":64,"value":2294},"Raw binary",{"type":64,"value":2296}," (fixed-shape)",{"type":58,"tag":134,"props":2298,"children":2299},{},[2300],{"type":58,"tag":79,"props":2301,"children":2303},{"className":2302},[],[2304],{"type":64,"value":2305},"arr = np.memmap(p, dtype=DTYPE, mode=\"r\", shape=(rows_in_file, *trailing_shape)); host = np.ascontiguousarray(arr[file_lo:file_hi])",{"type":58,"tag":112,"props":2307,"children":2308},{},[2309,2317],{"type":58,"tag":134,"props":2310,"children":2311},{},[2312],{"type":58,"tag":71,"props":2313,"children":2314},{},[2315],{"type":64,"value":2316},"HDF5",{"type":58,"tag":134,"props":2318,"children":2319},{},[2320],{"type":58,"tag":79,"props":2321,"children":2323},{"className":2322},[],[2324],{"type":64,"value":2325},"with h5py.File(p, \"r\") as f: host = np.ascontiguousarray(f[\"data\"][file_lo:file_hi])",{"type":58,"tag":112,"props":2327,"children":2328},{},[2329,2337],{"type":58,"tag":134,"props":2330,"children":2331},{},[2332],{"type":58,"tag":71,"props":2333,"children":2334},{},[2335],{"type":64,"value":2336},"Parquet \u002F Arrow",{"type":58,"tag":134,"props":2338,"children":2339},{},[2340],{"type":58,"tag":79,"props":2341,"children":2343},{"className":2342},[],[2344],{"type":64,"value":2345},"tbl = pq.read_table(p, columns=..., use_threads=False).slice(file_lo, file_hi - file_lo); host = tbl.to_pandas().values",{"type":58,"tag":67,"props":2347,"children":2348},{},[2349],{"type":64,"value":2350},"(For built-in single-call loaders per format, see the \"Why this skill\nexists\" table at the top of this file.)",{"type":58,"tag":67,"props":2352,"children":2353},{},[2354,2356,2361],{"type":64,"value":2355},"The discovery step (step 1) parses each format's metadata: ",{"type":58,"tag":79,"props":2357,"children":2359},{"className":2358},[],[2360],{"type":64,"value":92},{"type":64,"value":2362}," \u002F\nHDF5 \u002F Parquet all carry per-file row count + dtype on disk.\nRaw binary doesn't — sidecar or derive from file size.",{"type":58,"tag":267,"props":2364,"children":2366},{"id":2365},"common-pitfalls",[2367],{"type":64,"value":2368},"Common pitfalls",{"type":58,"tag":827,"props":2370,"children":2372},{"id":2371},"cnasarraydst-is-illegal-in-a-leaf-task",[2373,2379],{"type":58,"tag":79,"props":2374,"children":2376},{"className":2375},[],[2377],{"type":64,"value":2378},"cn.asarray(dst)",{"type":64,"value":2380}," is illegal in a leaf task",{"type":58,"tag":67,"props":2382,"children":2383},{},[2384,2386,2392,2394,2400,2402,2408,2410,2416],{"type":64,"value":2385},"Inside a ",{"type":58,"tag":79,"props":2387,"children":2389},{"className":2388},[],[2390],{"type":64,"value":2391},"@task",{"type":64,"value":2393}," body, any cupynumeric op that touches the top-level\nruntime — ",{"type":58,"tag":79,"props":2395,"children":2397},{"className":2396},[],[2398],{"type":64,"value":2399},"cn.asarray(store)",{"type":64,"value":2401},", slice assignment ",{"type":58,"tag":79,"props":2403,"children":2405},{"className":2404},[],[2406],{"type":64,"value":2407},"cn_dst[s] = host_np",{"type":64,"value":2409}," —\ntriggers ",{"type":58,"tag":79,"props":2411,"children":2413},{"className":2412},[],[2414],{"type":64,"value":2415},"create_index_space",{"type":64,"value":2417}," from the wrong context and Legion aborts:",{"type":58,"tag":454,"props":2419,"children":2423},{"className":2420,"code":2422,"language":64},[2421],"language-text","LEGION API USAGE EXCEPTION: Invalid task context passed to runtime call\ncreate_index_space\n",[2424],{"type":58,"tag":79,"props":2425,"children":2426},{"__ignoreMap":459},[2427],{"type":64,"value":2422},{"type":58,"tag":67,"props":2429,"children":2430},{},[2431,2433,2438,2440,2446,2448,2454],{"type":64,"value":2432},"Fix: consume the DLPack capsule with a ",{"type":58,"tag":71,"props":2434,"children":2435},{},[2436],{"type":64,"value":2437},"third-party",{"type":64,"value":2439}," library (cupy \u002F\ntorch \u002F numpy) inside leaf tasks. ",{"type":58,"tag":79,"props":2441,"children":2443},{"className":2442},[],[2444],{"type":64,"value":2445},"cn.asarray",{"type":64,"value":2447}," is fine in the driver,\njust not in leaf tasks. See ",{"type":58,"tag":79,"props":2449,"children":2451},{"className":2450},[],[2452],{"type":64,"value":2453},"examples\u002Fdlpack\u002Fleaf_task_interop.py",{"type":64,"value":2455}," for\nthe torch-flavoured workaround.",{"type":58,"tag":827,"props":2457,"children":2459},{"id":2458},"in-task-assert-aborts-the-runtime",[2460,2462,2468],{"type":64,"value":2461},"In-task ",{"type":58,"tag":79,"props":2463,"children":2465},{"className":2464},[],[2466],{"type":64,"value":2467},"assert",{"type":64,"value":2469}," aborts the runtime",{"type":58,"tag":67,"props":2471,"children":2472},{},[2473,2475,2480,2482,2488],{"type":64,"value":2474},"Legate treats unraised exceptions in a ",{"type":58,"tag":79,"props":2476,"children":2478},{"className":2477},[],[2479],{"type":64,"value":2391},{"type":64,"value":2481}," as a contract violation\nand aborts unless the task was registered with ",{"type":58,"tag":79,"props":2483,"children":2485},{"className":2484},[],[2486],{"type":64,"value":2487},"throws_exception()",{"type":64,"value":2489},".\nSanity-check on the host before launching.",{"type":58,"tag":827,"props":2491,"children":2493},{"id":2492},"launch-domain-must-match-the-partition-tile-count",[2494],{"type":64,"value":2495},"Launch domain must match the partition tile count",{"type":58,"tag":67,"props":2497,"children":2498},{},[2499,2505,2506,2512,2514,2520,2522,2528,2530,2536,2538,2544],{"type":58,"tag":79,"props":2500,"children":2502},{"className":2501},[],[2503],{"type":64,"value":2504},"create_manual_task(launch_shape=...)",{"type":64,"value":1032},{"type":58,"tag":79,"props":2507,"children":2509},{"className":2508},[],[2510],{"type":64,"value":2511},"partition_by_tiling(...)",{"type":64,"value":2513},"\nare independent — the runtime doesn't catch a mismatch. Larger launch\ndomain → out-of-range tiles; smaller → unwritten tiles. Always derive\nboth from the same ",{"type":58,"tag":79,"props":2515,"children":2517},{"className":2516},[],[2518],{"type":64,"value":2519},"(total_rows, tile_rows)",{"type":64,"value":2521}," via two separate ",{"type":58,"tag":79,"props":2523,"children":2525},{"className":2524},[],[2526],{"type":64,"value":2527},"ceil",{"type":64,"value":2529},"\ndivisions (sizing the launch domain to ",{"type":58,"tag":79,"props":2531,"children":2533},{"className":2532},[],[2534],{"type":64,"value":2535},"num_processors",{"type":64,"value":2537}," directly\nwould over-launch when ",{"type":58,"tag":79,"props":2539,"children":2541},{"className":2540},[],[2542],{"type":64,"value":2543},"num_processors > total_rows",{"type":64,"value":2545},"):",{"type":58,"tag":454,"props":2547,"children":2549},{"className":855,"code":2548,"language":857,"meta":459,"style":459},"tile_rows = max(1, (total_rows + num_processors - 1) \u002F\u002F num_processors)\nnum_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows\npartition = ...partition_by_tiling((tile_rows,) + trailing_shape)\nruntime.create_manual_task(load_tile.library, load_tile.task_id, (num_tasks,))\n",[2550],{"type":58,"tag":79,"props":2551,"children":2552},{"__ignoreMap":459},[2553,2560,2568,2576],{"type":58,"tag":465,"props":2554,"children":2555},{"class":467,"line":468},[2556],{"type":58,"tag":465,"props":2557,"children":2558},{},[2559],{"type":64,"value":1307},{"type":58,"tag":465,"props":2561,"children":2562},{"class":467,"line":478},[2563],{"type":58,"tag":465,"props":2564,"children":2565},{},[2566],{"type":64,"value":2567},"num_tasks = (total_rows + tile_rows - 1) \u002F\u002F tile_rows\n",{"type":58,"tag":465,"props":2569,"children":2570},{"class":467,"line":525},[2571],{"type":58,"tag":465,"props":2572,"children":2573},{},[2574],{"type":64,"value":2575},"partition = ...partition_by_tiling((tile_rows,) + trailing_shape)\n",{"type":58,"tag":465,"props":2577,"children":2578},{"class":467,"line":538},[2579],{"type":58,"tag":465,"props":2580,"children":2581},{},[2582],{"type":64,"value":2583},"runtime.create_manual_task(load_tile.library, load_tile.task_id, (num_tasks,))\n",{"type":58,"tag":2585,"props":2586,"children":2587},"style",{},[2588],{"type":64,"value":2589},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2591,"total":2749},[2592,2610,2627,2638,2650,2664,2677,2691,2704,2715,2729,2738],{"slug":2593,"name":2593,"fn":2594,"description":2595,"org":2596,"tags":2597,"stars":2607,"repoUrl":2608,"updatedAt":2609},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2598,2601,2604],{"name":2599,"slug":2600,"type":15},"Documentation","documentation",{"name":2602,"slug":2603,"type":15},"MCP","mcp",{"name":2605,"slug":2606,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2611,"name":2611,"fn":2612,"description":2613,"org":2614,"tags":2615,"stars":2624,"repoUrl":2625,"updatedAt":2626},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2616,2619,2622],{"name":2617,"slug":2618,"type":15},"Containers","containers",{"name":2620,"slug":2621,"type":15},"Deployment","deployment",{"name":2623,"slug":857,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2628,"name":2628,"fn":2629,"description":2630,"org":2631,"tags":2632,"stars":2624,"repoUrl":2625,"updatedAt":2637},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2633,2636],{"name":2634,"slug":2635,"type":15},"CI\u002FCD","ci-cd",{"name":2620,"slug":2621,"type":15},"2026-07-14T05:25:59.97109",{"slug":2639,"name":2639,"fn":2640,"description":2641,"org":2642,"tags":2643,"stars":2624,"repoUrl":2625,"updatedAt":2649},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2644,2645,2646],{"name":2634,"slug":2635,"type":15},{"name":2620,"slug":2621,"type":15},{"name":2647,"slug":2648,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2651,"name":2651,"fn":2652,"description":2653,"org":2654,"tags":2655,"stars":2624,"repoUrl":2625,"updatedAt":2663},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2656,2659,2660],{"name":2657,"slug":2658,"type":15},"Debugging","debugging",{"name":2647,"slug":2648,"type":15},{"name":2661,"slug":2662,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2665,"name":2665,"fn":2666,"description":2667,"org":2668,"tags":2669,"stars":2624,"repoUrl":2625,"updatedAt":2676},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2670,2673],{"name":2671,"slug":2672,"type":15},"Best Practices","best-practices",{"name":2674,"slug":2675,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2678,"name":2678,"fn":2679,"description":2680,"org":2681,"tags":2682,"stars":2624,"repoUrl":2625,"updatedAt":2690},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2683,2686,2689],{"name":2684,"slug":2685,"type":15},"Machine Learning","machine-learning",{"name":2687,"slug":2688,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2692,"name":2692,"fn":2693,"description":2694,"org":2695,"tags":2696,"stars":2624,"repoUrl":2625,"updatedAt":2703},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2697,2700],{"name":2698,"slug":2699,"type":15},"QA","qa",{"name":2701,"slug":2702,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2705,"name":2705,"fn":2706,"description":2707,"org":2708,"tags":2709,"stars":2624,"repoUrl":2625,"updatedAt":2714},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2710,2711],{"name":2620,"slug":2621,"type":15},{"name":2712,"slug":2713,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2716,"name":2716,"fn":2717,"description":2718,"org":2719,"tags":2720,"stars":2624,"repoUrl":2625,"updatedAt":2728},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2721,2724,2725],{"name":2722,"slug":2723,"type":15},"Code Review","code-review",{"name":2647,"slug":2648,"type":15},{"name":2726,"slug":2727,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2730,"name":2730,"fn":2731,"description":2732,"org":2733,"tags":2734,"stars":2624,"repoUrl":2625,"updatedAt":2737},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2735,2736],{"name":2698,"slug":2699,"type":15},{"name":2701,"slug":2702,"type":15},"2026-07-14T05:25:54.928983",{"slug":2739,"name":2739,"fn":2740,"description":2741,"org":2742,"tags":2743,"stars":2624,"repoUrl":2625,"updatedAt":2748},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2744,2747],{"name":2745,"slug":2746,"type":15},"Automation","automation",{"name":2634,"slug":2635,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2751,"total":2841},[2752,2763,2773,2787,2797,2812,2827],{"slug":2753,"name":2753,"fn":2754,"description":2755,"org":2756,"tags":2757,"stars":26,"repoUrl":27,"updatedAt":2762},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2758,2759,2760,2761],{"name":23,"slug":24,"type":15},{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":2764,"name":2764,"fn":2765,"description":2766,"org":2767,"tags":2768,"stars":26,"repoUrl":27,"updatedAt":2772},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2769,2770,2771],{"name":2620,"slug":2621,"type":15},{"name":2712,"slug":2713,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2774,"name":2774,"fn":2775,"description":2776,"org":2777,"tags":2778,"stars":26,"repoUrl":27,"updatedAt":2786},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2779,2782,2783],{"name":2780,"slug":2781,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2784,"slug":2785,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2788,"name":2788,"fn":2789,"description":2790,"org":2791,"tags":2792,"stars":26,"repoUrl":27,"updatedAt":2796},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2793,2794,2795],{"name":23,"slug":24,"type":15},{"name":9,"slug":8,"type":15},{"name":2701,"slug":2702,"type":15},"2026-07-17T05:29:03.913266",{"slug":2798,"name":2798,"fn":2799,"description":2800,"org":2801,"tags":2802,"stars":26,"repoUrl":27,"updatedAt":2811},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2803,2804,2807,2808],{"name":2745,"slug":2746,"type":15},{"name":2805,"slug":2806,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2809,"slug":2810,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2813,"name":2813,"fn":2814,"description":2815,"org":2816,"tags":2817,"stars":26,"repoUrl":27,"updatedAt":2826},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2818,2819,2822,2823],{"name":2620,"slug":2621,"type":15},{"name":2820,"slug":2821,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2824,"slug":2825,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2828,"name":2828,"fn":2829,"description":2830,"org":2831,"tags":2832,"stars":26,"repoUrl":27,"updatedAt":2840},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2833,2834,2837],{"name":9,"slug":8,"type":15},{"name":2835,"slug":2836,"type":15},"Quantum Computing","quantum-computing",{"name":2838,"slug":2839,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]