[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-replicate-build-models":3,"mdc--adgxqz-key":33,"related-org-replicate-build-models":2652,"related-repo-replicate-build-models":2732},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":28,"sourceUrl":31,"mdContent":32},"build-models","build and package custom AI models","Package and build custom AI models with Cog for deployment on Replicate. Use when creating a cog.yaml or predict.py, defining model inputs and outputs, loading model weights at setup time, building Docker images for ML models, serving locally with cog serve or cog predict, or porting a HuggingFace, GitHub, or ComfyUI model to run on Replicate. Trigger on phrases like \"build a model\", \"package a model\", \"create a Cog model\", \"wrap a model\", \"containerize an AI model\", \"predict.py\", \"cog.yaml\", \"BasePredictor\", or \"Cog container\", and when referencing cog.run, github.com\u002Freplicate\u002Fcog, or github.com\u002Freplicate\u002Fcog-examples. Covers GPU and CUDA setup, pget for fast weight downloads, async predictors with continuous batching, streaming outputs, and cold-boot optimization for image, video, audio, and LLM models. For pushing built models to Replicate, see publish-models. For running existing models, see run-models.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"replicate","Replicate","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Freplicate.jpg",[12,16,19],{"name":13,"slug":14,"type":15},"Machine Learning","machine-learning","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":20,"slug":21,"type":15},"AI Infrastructure","ai-infrastructure",51,"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fskills","2026-07-16T06:02:28.318441",null,6,[],{"repoUrl":23,"stars":22,"forks":26,"topics":29,"description":30},[],"A collection of Agent Skills for building AI-powered apps with Replicate","https:\u002F\u002Fgithub.com\u002Freplicate\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fbuild-models","---\nname: build-models\ndescription: >\n  Package and build custom AI models with Cog for deployment on Replicate.\n  Use when creating a cog.yaml or predict.py, defining model inputs and\n  outputs, loading model weights at setup time, building Docker images for\n  ML models, serving locally with cog serve or cog predict, or porting a\n  HuggingFace, GitHub, or ComfyUI model to run on Replicate. Trigger on\n  phrases like \"build a model\", \"package a model\", \"create a Cog model\",\n  \"wrap a model\", \"containerize an AI model\", \"predict.py\", \"cog.yaml\",\n  \"BasePredictor\", or \"Cog container\", and when referencing cog.run,\n  github.com\u002Freplicate\u002Fcog, or github.com\u002Freplicate\u002Fcog-examples. Covers\n  GPU and CUDA setup, pget for fast weight downloads, async predictors\n  with continuous batching, streaming outputs, and cold-boot optimization\n  for image, video, audio, and LLM models. For pushing built models to\n  Replicate, see publish-models. For running existing models, see\n  run-models.\n---\n\n## Docs\n\n- Cog reference (single file): \u003Chttps:\u002F\u002Fcog.run\u002Fllms.txt>\n- `cog.yaml` reference: \u003Chttps:\u002F\u002Fcog.run\u002Fyaml>\n- Python predictor reference: \u003Chttps:\u002F\u002Fcog.run\u002Fpython>\n- Examples: \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-examples>\n- Template: \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-template>\n\n## When to use this skill\n\n- You have model code, weights, or a HuggingFace\u002FGitHub project you want to host on Replicate.\n- You're writing or editing a `cog.yaml`, `predict.py`, or `train.py`.\n- For pushing a built model to Replicate, see `publish-models`.\n- For running existing Replicate models, see `run-models`.\n\n## Prerequisites\n\n- Docker running locally.\n- Cog installed: `brew install replicate\u002Ftap\u002Fcog` or `sh \u003C(curl -fsSL https:\u002F\u002Fcog.run\u002Finstall.sh)`.\n- Optional: `cog init` to scaffold `cog.yaml` and `predict.py`.\n\n## Project layout\n\nThe canonical Replicate model layout:\n\n```\ncog.yaml\npredict.py\nweights.py                 # optional download helpers\nrequirements.txt\ncog-safe-push-configs\u002F\n  default.yaml             # see publish-models skill\n.github\u002Fworkflows\u002F\n  ci.yaml\nscript\u002F                    # github.com\u002Fgithub\u002Fscripts-to-rule-them-all\n  lint\n  test\n  push\n```\n\n## cog.yaml essentials\n\nA modern config for a GPU model:\n\n```yaml\nbuild:\n  gpu: true\n  cuda: \"12.8\"\n  python_version: \"3.12\"\n  python_requirements: requirements.txt\n  system_packages:\n    - libgl1\n    - libglib2.0-0\npredict: predict.py:Predictor\n```\n\nNotes:\n\n- Pin Python to a specific minor version, and pin every line in `requirements.txt`. Floating versions break cold boots.\n- Use `python_requirements` over inline `python_packages` once the list grows.\n- `cuda` follows your torch wheel (e.g. `12.8` paired with `torch==2.7.1+cu128`).\n- Add `train: train.py:train` if your model is fine-tunable.\n- Add `image: r8.im\u002Fowner\u002Fname` to enable bare `cog push`.\n\nFor async predictors with continuous batching:\n\n```yaml\nconcurrency:\n  max: 32\n```\n\n## predict.py essentials\n\n```python\nfrom cog import BasePredictor, Input, Path\n\nclass Predictor(BasePredictor):\n    def setup(self) -> None:\n        \"\"\"One-time loads. Heavy work goes here, not in predict().\"\"\"\n        self.model = load_model(\"weights\u002F\")\n\n    def predict(\n        self,\n        prompt: str = Input(description=\"Text prompt for generation\"),\n        seed: int = Input(description=\"Random seed; leave blank for random\", default=None),\n        num_steps: int = Input(description=\"Number of denoising steps\", ge=1, le=50, default=20),\n        output_format: str = Input(description=\"Output image format\", choices=[\"webp\", \"jpg\", \"png\"], default=\"webp\"),\n    ) -> Path:\n        \"\"\"Run a single prediction.\"\"\"\n        if not prompt.strip():\n            raise ValueError(\"prompt cannot be empty\")\n        out = self.model.generate(prompt, seed=seed, steps=num_steps)\n        return Path(out)\n```\n\nInput rules:\n\n- Every input needs a `description`. The description shows up in the model schema and on Replicate's web UI.\n- Use `ge`\u002F`le` for numeric bounds, `choices=[...]` for enums, `regex=` for strings.\n- Use `cog.Path` for file inputs and outputs, never raw bytes.\n- Use `cog.Secret` for any token-like input (HF tokens, API keys), never plain `str`.\n- Provide a default that's inside `choices` for categorical inputs.\n- Validate inputs early in `predict()` and raise `ValueError`.\n\nStreaming text output (for LLMs):\n\n```python\nfrom cog import BasePredictor, Input, ConcatenateIterator\n\nclass Predictor(BasePredictor):\n    def predict(self, prompt: str = Input(description=\"Prompt\")) -> ConcatenateIterator[str]:\n        for token in self.model.stream(prompt):\n            yield token\n```\n\nAsync predictor with continuous batching (paired with `concurrency.max` in cog.yaml):\n\n```python\nfrom cog import BasePredictor, Input, AsyncConcatenateIterator\n\nclass Predictor(BasePredictor):\n    async def setup(self) -> None:\n        self.engine = await load_async_engine()\n\n    async def predict(\n        self,\n        prompt: str = Input(description=\"Prompt\"),\n    ) -> AsyncConcatenateIterator[str]:\n        async for token in self.engine.generate(prompt):\n            yield token\n```\n\nDynamic `choices` from on-disk assets (e.g. a `voices\u002F` directory of audio samples):\n\n```python\nfrom pathlib import Path as _P\nAVAILABLE_VOICES = sorted(p.stem for p in _P(\"voices\").glob(\"*.wav\"))\n\nclass Predictor(BasePredictor):\n    def predict(\n        self,\n        speaker: str = Input(description=\"Voice\", choices=AVAILABLE_VOICES, default=AVAILABLE_VOICES[0]),\n    ) -> Path: ...\n```\n\n## Loading weights fast\n\nCold boot dominates user-perceived latency. Three patterns, ranked by simplicity:\n\n### 1. Bake weights into the image at build time\n\nBest for small or medium weights (\u003C 5GB) that you want zero-cold-boot for.\n\nFor torchvision:\n\n```python\nimport os\nos.environ[\"TORCH_HOME\"] = \".\"  # set before importing torch\nimport torch\nfrom torchvision import models\n```\n\nFor HuggingFace:\n\n```python\nimport os\nos.environ[\"HF_HUB_CACHE\"] = \".\u002F.cache\"\nos.environ[\"HF_XET_HIGH_PERFORMANCE\"] = \"1\"\n```\n\nThen download once during `cog build` (e.g. in a `run:` step or by running a small fetcher script as part of the build). The weights become part of the image layer.\n\n### 2. Pull from `weights.replicate.delivery` with pget\n\nBest for large weights, or when you want to share weights across multiple models. `pget` is Replicate's parallel HTTP fetcher.\n\nIn `cog.yaml`:\n\n```yaml\nbuild:\n  run:\n    - curl -o \u002Fusr\u002Flocal\u002Fbin\u002Fpget -L \"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fpget\u002Freleases\u002Fdownload\u002Fv0.8.2\u002Fpget_linux_x86_64\"\n    - chmod +x \u002Fusr\u002Flocal\u002Fbin\u002Fpget\n```\n\nIn `setup()`:\n\n```python\nimport subprocess\nfrom pathlib import Path\n\nWEIGHTS_URL = \"https:\u002F\u002Fweights.replicate.delivery\u002Fdefault\u002Fmy-model\u002Fweights.tar\"\nWEIGHTS_DIR = Path(\"weights\")\n\nclass Predictor(BasePredictor):\n    def setup(self) -> None:\n        if not WEIGHTS_DIR.exists():\n            # -x extracts tar in-memory; default concurrency is 4 * NumCPU\n            subprocess.check_call([\"pget\", \"-x\", WEIGHTS_URL, str(WEIGHTS_DIR)])\n        self.model = load_from(WEIGHTS_DIR)\n```\n\nFor multiple files in one shot:\n\n```python\nmanifest = \"\\n\".join([\n    f\"{base}\u002Funet.safetensors weights\u002Funet.safetensors\",\n    f\"{base}\u002Fvae.safetensors  weights\u002Fvae.safetensors\",\n    f\"{base}\u002Ftext_encoder.safetensors weights\u002Ftext_encoder.safetensors\",\n])\nsubprocess.run([\"pget\", \"multifile\", \"-\"], input=manifest, text=True, check=True)\n```\n\n### 3. HuggingFace Hub with hf_transfer\n\nSet `HF_HUB_ENABLE_HF_TRANSFER=1` and use `huggingface_hub.snapshot_download` or `from_pretrained`. Faster than vanilla HF downloads. Use a `cog.Secret` input for gated models.\n\n## Weight cache for user-supplied weights\n\nFor LoRAs or any weights URL the user passes at predict time, use a sha256-keyed disk cache with LRU eviction:\n\n```python\nimport hashlib, shutil, subprocess\nfrom pathlib import Path\n\nclass WeightsDownloadCache:\n    def __init__(self, cache_dir: str = \"\u002Ftmp\u002Fweights-cache\", min_disk_free_gb: int = 10):\n        self.cache_dir = Path(cache_dir)\n        self.cache_dir.mkdir(parents=True, exist_ok=True)\n        self.min_disk_free = min_disk_free_gb * 1024**3\n\n    def ensure(self, url: str) -> Path:\n        key = hashlib.sha256(url.encode()).hexdigest()\n        target = self.cache_dir \u002F key\n        if target.exists():\n            target.touch()  # bump LRU mtime\n            return target\n        self._evict_until_room()\n        subprocess.check_call([\"pget\", url, str(target)])\n        return target\n\n    def _evict_until_room(self) -> None:\n        while shutil.disk_usage(self.cache_dir).free \u003C self.min_disk_free:\n            entries = sorted(self.cache_dir.iterdir(), key=lambda p: p.stat().st_mtime)\n            if not entries:\n                return\n            entries[0].unlink()\n```\n\nSee `replicate\u002Fcog-flux\u002Fweights.py` for a production version that handles HF, CivitAI, Replicate, and arbitrary `.safetensors` URLs.\n\n## Multi-LoRA composition\n\nReload only when the URL changes; compose two LoRAs with separate scales:\n\n```python\nclass Predictor(BasePredictor):\n    def setup(self) -> None:\n        self.pipe = load_base_pipeline()\n        self.loaded = {\"main\": None, \"extra\": None}\n\n    def _ensure_lora(self, slot: str, url: str | None) -> None:\n        if url == self.loaded[slot]:\n            return\n        if self.loaded[slot] is not None:\n            self.pipe.unload_lora_weights(adapter_name=slot)\n        if url:\n            path = self.cache.ensure(url)\n            self.pipe.load_lora_weights(str(path), adapter_name=slot)\n        self.loaded[slot] = url\n\n    def predict(\n        self,\n        prompt: str = Input(description=\"Prompt\"),\n        lora_url: str = Input(description=\"Primary LoRA URL\", default=None),\n        lora_scale: float = Input(description=\"Primary LoRA scale\", ge=0.0, le=2.0, default=1.0),\n        extra_lora_url: str = Input(description=\"Optional second LoRA URL\", default=None),\n        extra_lora_scale: float = Input(description=\"Second LoRA scale\", ge=0.0, le=2.0, default=1.0),\n    ) -> Path:\n        self._ensure_lora(\"main\", lora_url)\n        self._ensure_lora(\"extra\", extra_lora_url)\n        adapters = [s for s, u in self.loaded.items() if u]\n        scales = [lora_scale if s == \"main\" else extra_lora_scale for s in adapters]\n        if adapters:\n            self.pipe.set_adapters(adapters, adapter_weights=scales)\n        return Path(self.pipe(prompt).images[0].save(\"\u002Ftmp\u002Fout.png\"))\n```\n\n## Cold-boot tricks\n\nFrom production diffusion models like `replicate\u002Fcog-flux` and `replicate\u002Fcog-flux-kontext`:\n\n- Set perf flags once in `setup()`:\n  ```python\n  import torch\n  torch.set_float32_matmul_precision(\"high\")\n  torch.backends.cuda.matmul.allow_tf32 = True\n  torch.backends.cudnn.benchmark = True\n  ```\n- Compile and warm up:\n  ```python\n  self.model = torch.compile(self.model, dynamic=True)\n  _ = self.predict(prompt=\"warmup\", num_steps=1)  # absorbs compile cost in setup\n  ```\n- Load big weights with meta device + `assign=True` to avoid double-allocating:\n  ```python\n  with torch.device(\"meta\"):\n      model = build_model_skeleton()\n  state = torch.load(\"weights.pt\", map_location=\"cpu\")\n  model.load_state_dict(state, assign=True)\n  ```\n- Share VAE \u002F text encoder across multiple pipelines (e.g. base + img2img + inpaint) instead of loading three copies.\n- For fp8\u002Fint8, save quantized weights ahead of time and load directly; don't quantize at boot.\n\n## Local development\n\n```\ncog init                                    # scaffold cog.yaml + predict.py\ncog predict -i prompt=\"hello\"               # build + run a single prediction\ncog predict -i image=@input.jpg -o out.png  # file inputs and outputs\ncog serve -p 8393                           # HTTP server matching production\ncog exec python                             # interactive shell inside the build env\n```\n\n## Building\n\n```\ncog build -t my-model\ncog build --separate-weights -t my-model    # weights in their own image layer\ncog build --secret id=hf,src=$HOME\u002F.hf_token -t my-model\n```\n\nTips:\n\n- Use `--separate-weights` for any model with weights > ~1GB. It speeds up cold boots and registry pushes.\n- Use `--mount=type=cache,target=\u002Froot\u002F.cache\u002Fpip` in `run:` steps to cache pip across builds.\n- Use `--secret` instead of `ARG` to keep tokens out of image history.\n- The default Cog base image (`--use-cog-base-image=true`) is faster than rolling your own.\n\n## Training\n\nIf your model supports fine-tuning, add `train: train.py:train` to `cog.yaml` and write a `train()` function that returns `TrainingOutput(weights=Path(\"model.tar\"))`. The predictor then accepts the URL via `setup(self, weights)` or the `COG_WEIGHTS` env var. See \u003Chttps:\u002F\u002Fcog.run\u002Ftraining> and `replicate\u002Fflux-fine-tuner` for a full example.\n\n## Guidelines\n\n- Keep `setup()` for one-time loads; keep `predict()` fast and deterministic in shape.\n- Pin Python and every dependency. Use `numpy\u003C2` if your torch is older.\n- Always describe every input. Schemas without descriptions are unusable on the web UI.\n- Use `cog.Path` for files and `cog.Secret` for tokens.\n- Pin `pget` to a specific release (`v0.8.2`) for reproducibility.\n- Set `HF_HUB_ENABLE_HF_TRANSFER=1` whenever you call HuggingFace Hub.\n- Set `TRANSFORMERS_OFFLINE=1` after weights are loaded to prevent runtime HF lookups.\n- Test with `cog predict` before pushing. If it doesn't work locally, it won't work in production.\n\n## Production references\n\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-examples> — minimal patterns (resnet, hello-world, streaming, training)\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-template> — scaffolder for new model repos\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-flux> — multi-variant FLUX models, weights cache, fp8 + torch.compile\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-flux-kontext> — meta-device loading, warmup compilation\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-vllm> — async LLM server with continuous batching, training-as-packaging\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-comfyui> — ComfyUI workflows as a Cog model, custom-node helpers\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fflux-fine-tuner> — multi-LoRA composition, shared pipeline components\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fvibevoice> — TTS with dynamic `choices`, minimal cog.yaml\n- \u003Chttps:\u002F\u002Fgithub.com\u002Freplicate\u002Fpget> — parallel weights fetcher\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,47,116,122,182,188,242,248,254,266,272,277,454,459,554,559,596,602,773,778,894,899,952,965,1063,1083,1150,1156,1161,1168,1173,1178,1217,1222,1252,1273,1287,1300,1311,1365,1376,1475,1480,1535,1541,1576,1582,1587,1796,1817,1823,1828,2072,2078,2097,2239,2245,2254,2260,2269,2274,2341,2347,2413,2419,2533,2539,2646],{"type":39,"tag":40,"props":41,"children":43},"element","h2",{"id":42},"docs",[44],{"type":45,"value":46},"text","Docs",{"type":39,"tag":48,"props":49,"children":50},"ul",{},[51,65,83,94,105],{"type":39,"tag":52,"props":53,"children":54},"li",{},[55,57],{"type":45,"value":56},"Cog reference (single file): ",{"type":39,"tag":58,"props":59,"children":63},"a",{"href":60,"rel":61},"https:\u002F\u002Fcog.run\u002Fllms.txt",[62],"nofollow",[64],{"type":45,"value":60},{"type":39,"tag":52,"props":66,"children":67},{},[68,75,77],{"type":39,"tag":69,"props":70,"children":72},"code",{"className":71},[],[73],{"type":45,"value":74},"cog.yaml",{"type":45,"value":76}," reference: ",{"type":39,"tag":58,"props":78,"children":81},{"href":79,"rel":80},"https:\u002F\u002Fcog.run\u002Fyaml",[62],[82],{"type":45,"value":79},{"type":39,"tag":52,"props":84,"children":85},{},[86,88],{"type":45,"value":87},"Python predictor reference: ",{"type":39,"tag":58,"props":89,"children":92},{"href":90,"rel":91},"https:\u002F\u002Fcog.run\u002Fpython",[62],[93],{"type":45,"value":90},{"type":39,"tag":52,"props":95,"children":96},{},[97,99],{"type":45,"value":98},"Examples: ",{"type":39,"tag":58,"props":100,"children":103},{"href":101,"rel":102},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-examples",[62],[104],{"type":45,"value":101},{"type":39,"tag":52,"props":106,"children":107},{},[108,110],{"type":45,"value":109},"Template: ",{"type":39,"tag":58,"props":111,"children":114},{"href":112,"rel":113},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-template",[62],[115],{"type":45,"value":112},{"type":39,"tag":40,"props":117,"children":119},{"id":118},"when-to-use-this-skill",[120],{"type":45,"value":121},"When to use this skill",{"type":39,"tag":48,"props":123,"children":124},{},[125,130,158,170],{"type":39,"tag":52,"props":126,"children":127},{},[128],{"type":45,"value":129},"You have model code, weights, or a HuggingFace\u002FGitHub project you want to host on Replicate.",{"type":39,"tag":52,"props":131,"children":132},{},[133,135,140,142,148,150,156],{"type":45,"value":134},"You're writing or editing a ",{"type":39,"tag":69,"props":136,"children":138},{"className":137},[],[139],{"type":45,"value":74},{"type":45,"value":141},", ",{"type":39,"tag":69,"props":143,"children":145},{"className":144},[],[146],{"type":45,"value":147},"predict.py",{"type":45,"value":149},", or ",{"type":39,"tag":69,"props":151,"children":153},{"className":152},[],[154],{"type":45,"value":155},"train.py",{"type":45,"value":157},".",{"type":39,"tag":52,"props":159,"children":160},{},[161,163,169],{"type":45,"value":162},"For pushing a built model to Replicate, see ",{"type":39,"tag":69,"props":164,"children":166},{"className":165},[],[167],{"type":45,"value":168},"publish-models",{"type":45,"value":157},{"type":39,"tag":52,"props":171,"children":172},{},[173,175,181],{"type":45,"value":174},"For running existing Replicate models, see ",{"type":39,"tag":69,"props":176,"children":178},{"className":177},[],[179],{"type":45,"value":180},"run-models",{"type":45,"value":157},{"type":39,"tag":40,"props":183,"children":185},{"id":184},"prerequisites",[186],{"type":45,"value":187},"Prerequisites",{"type":39,"tag":48,"props":189,"children":190},{},[191,196,216],{"type":39,"tag":52,"props":192,"children":193},{},[194],{"type":45,"value":195},"Docker running locally.",{"type":39,"tag":52,"props":197,"children":198},{},[199,201,207,209,215],{"type":45,"value":200},"Cog installed: ",{"type":39,"tag":69,"props":202,"children":204},{"className":203},[],[205],{"type":45,"value":206},"brew install replicate\u002Ftap\u002Fcog",{"type":45,"value":208}," or ",{"type":39,"tag":69,"props":210,"children":212},{"className":211},[],[213],{"type":45,"value":214},"sh \u003C(curl -fsSL https:\u002F\u002Fcog.run\u002Finstall.sh)",{"type":45,"value":157},{"type":39,"tag":52,"props":217,"children":218},{},[219,221,227,229,234,236,241],{"type":45,"value":220},"Optional: ",{"type":39,"tag":69,"props":222,"children":224},{"className":223},[],[225],{"type":45,"value":226},"cog init",{"type":45,"value":228}," to scaffold ",{"type":39,"tag":69,"props":230,"children":232},{"className":231},[],[233],{"type":45,"value":74},{"type":45,"value":235}," and ",{"type":39,"tag":69,"props":237,"children":239},{"className":238},[],[240],{"type":45,"value":147},{"type":45,"value":157},{"type":39,"tag":40,"props":243,"children":245},{"id":244},"project-layout",[246],{"type":45,"value":247},"Project layout",{"type":39,"tag":249,"props":250,"children":251},"p",{},[252],{"type":45,"value":253},"The canonical Replicate model layout:",{"type":39,"tag":255,"props":256,"children":260},"pre",{"className":257,"code":259,"language":45},[258],"language-text","cog.yaml\npredict.py\nweights.py                 # optional download helpers\nrequirements.txt\ncog-safe-push-configs\u002F\n  default.yaml             # see publish-models skill\n.github\u002Fworkflows\u002F\n  ci.yaml\nscript\u002F                    # github.com\u002Fgithub\u002Fscripts-to-rule-them-all\n  lint\n  test\n  push\n",[261],{"type":39,"tag":69,"props":262,"children":264},{"__ignoreMap":263},"",[265],{"type":45,"value":259},{"type":39,"tag":40,"props":267,"children":269},{"id":268},"cogyaml-essentials",[270],{"type":45,"value":271},"cog.yaml essentials",{"type":39,"tag":249,"props":273,"children":274},{},[275],{"type":45,"value":276},"A modern config for a GPU model:",{"type":39,"tag":255,"props":278,"children":282},{"className":279,"code":280,"language":281,"meta":263,"style":263},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","build:\n  gpu: true\n  cuda: \"12.8\"\n  python_version: \"3.12\"\n  python_requirements: requirements.txt\n  system_packages:\n    - libgl1\n    - libglib2.0-0\npredict: predict.py:Predictor\n","yaml",[283],{"type":39,"tag":69,"props":284,"children":285},{"__ignoreMap":263},[286,304,324,353,379,397,409,423,436],{"type":39,"tag":287,"props":288,"children":291},"span",{"class":289,"line":290},"line",1,[292,298],{"type":39,"tag":287,"props":293,"children":295},{"style":294},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[296],{"type":45,"value":297},"build",{"type":39,"tag":287,"props":299,"children":301},{"style":300},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[302],{"type":45,"value":303},":\n",{"type":39,"tag":287,"props":305,"children":307},{"class":289,"line":306},2,[308,313,318],{"type":39,"tag":287,"props":309,"children":310},{"style":294},[311],{"type":45,"value":312},"  gpu",{"type":39,"tag":287,"props":314,"children":315},{"style":300},[316],{"type":45,"value":317},":",{"type":39,"tag":287,"props":319,"children":321},{"style":320},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[322],{"type":45,"value":323}," true\n",{"type":39,"tag":287,"props":325,"children":327},{"class":289,"line":326},3,[328,333,337,342,348],{"type":39,"tag":287,"props":329,"children":330},{"style":294},[331],{"type":45,"value":332},"  cuda",{"type":39,"tag":287,"props":334,"children":335},{"style":300},[336],{"type":45,"value":317},{"type":39,"tag":287,"props":338,"children":339},{"style":300},[340],{"type":45,"value":341}," \"",{"type":39,"tag":287,"props":343,"children":345},{"style":344},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[346],{"type":45,"value":347},"12.8",{"type":39,"tag":287,"props":349,"children":350},{"style":300},[351],{"type":45,"value":352},"\"\n",{"type":39,"tag":287,"props":354,"children":356},{"class":289,"line":355},4,[357,362,366,370,375],{"type":39,"tag":287,"props":358,"children":359},{"style":294},[360],{"type":45,"value":361},"  python_version",{"type":39,"tag":287,"props":363,"children":364},{"style":300},[365],{"type":45,"value":317},{"type":39,"tag":287,"props":367,"children":368},{"style":300},[369],{"type":45,"value":341},{"type":39,"tag":287,"props":371,"children":372},{"style":344},[373],{"type":45,"value":374},"3.12",{"type":39,"tag":287,"props":376,"children":377},{"style":300},[378],{"type":45,"value":352},{"type":39,"tag":287,"props":380,"children":382},{"class":289,"line":381},5,[383,388,392],{"type":39,"tag":287,"props":384,"children":385},{"style":294},[386],{"type":45,"value":387},"  python_requirements",{"type":39,"tag":287,"props":389,"children":390},{"style":300},[391],{"type":45,"value":317},{"type":39,"tag":287,"props":393,"children":394},{"style":344},[395],{"type":45,"value":396}," requirements.txt\n",{"type":39,"tag":287,"props":398,"children":399},{"class":289,"line":26},[400,405],{"type":39,"tag":287,"props":401,"children":402},{"style":294},[403],{"type":45,"value":404},"  system_packages",{"type":39,"tag":287,"props":406,"children":407},{"style":300},[408],{"type":45,"value":303},{"type":39,"tag":287,"props":410,"children":412},{"class":289,"line":411},7,[413,418],{"type":39,"tag":287,"props":414,"children":415},{"style":300},[416],{"type":45,"value":417},"    -",{"type":39,"tag":287,"props":419,"children":420},{"style":344},[421],{"type":45,"value":422}," libgl1\n",{"type":39,"tag":287,"props":424,"children":426},{"class":289,"line":425},8,[427,431],{"type":39,"tag":287,"props":428,"children":429},{"style":300},[430],{"type":45,"value":417},{"type":39,"tag":287,"props":432,"children":433},{"style":344},[434],{"type":45,"value":435}," libglib2.0-0\n",{"type":39,"tag":287,"props":437,"children":439},{"class":289,"line":438},9,[440,445,449],{"type":39,"tag":287,"props":441,"children":442},{"style":294},[443],{"type":45,"value":444},"predict",{"type":39,"tag":287,"props":446,"children":447},{"style":300},[448],{"type":45,"value":317},{"type":39,"tag":287,"props":450,"children":451},{"style":344},[452],{"type":45,"value":453}," predict.py:Predictor\n",{"type":39,"tag":249,"props":455,"children":456},{},[457],{"type":45,"value":458},"Notes:",{"type":39,"tag":48,"props":460,"children":461},{},[462,475,496,522,535],{"type":39,"tag":52,"props":463,"children":464},{},[465,467,473],{"type":45,"value":466},"Pin Python to a specific minor version, and pin every line in ",{"type":39,"tag":69,"props":468,"children":470},{"className":469},[],[471],{"type":45,"value":472},"requirements.txt",{"type":45,"value":474},". Floating versions break cold boots.",{"type":39,"tag":52,"props":476,"children":477},{},[478,480,486,488,494],{"type":45,"value":479},"Use ",{"type":39,"tag":69,"props":481,"children":483},{"className":482},[],[484],{"type":45,"value":485},"python_requirements",{"type":45,"value":487}," over inline ",{"type":39,"tag":69,"props":489,"children":491},{"className":490},[],[492],{"type":45,"value":493},"python_packages",{"type":45,"value":495}," once the list grows.",{"type":39,"tag":52,"props":497,"children":498},{},[499,505,507,512,514,520],{"type":39,"tag":69,"props":500,"children":502},{"className":501},[],[503],{"type":45,"value":504},"cuda",{"type":45,"value":506}," follows your torch wheel (e.g. ",{"type":39,"tag":69,"props":508,"children":510},{"className":509},[],[511],{"type":45,"value":347},{"type":45,"value":513}," paired with ",{"type":39,"tag":69,"props":515,"children":517},{"className":516},[],[518],{"type":45,"value":519},"torch==2.7.1+cu128",{"type":45,"value":521},").",{"type":39,"tag":52,"props":523,"children":524},{},[525,527,533],{"type":45,"value":526},"Add ",{"type":39,"tag":69,"props":528,"children":530},{"className":529},[],[531],{"type":45,"value":532},"train: train.py:train",{"type":45,"value":534}," if your model is fine-tunable.",{"type":39,"tag":52,"props":536,"children":537},{},[538,539,545,547,553],{"type":45,"value":526},{"type":39,"tag":69,"props":540,"children":542},{"className":541},[],[543],{"type":45,"value":544},"image: r8.im\u002Fowner\u002Fname",{"type":45,"value":546}," to enable bare ",{"type":39,"tag":69,"props":548,"children":550},{"className":549},[],[551],{"type":45,"value":552},"cog push",{"type":45,"value":157},{"type":39,"tag":249,"props":555,"children":556},{},[557],{"type":45,"value":558},"For async predictors with continuous batching:",{"type":39,"tag":255,"props":560,"children":562},{"className":279,"code":561,"language":281,"meta":263,"style":263},"concurrency:\n  max: 32\n",[563],{"type":39,"tag":69,"props":564,"children":565},{"__ignoreMap":263},[566,578],{"type":39,"tag":287,"props":567,"children":568},{"class":289,"line":290},[569,574],{"type":39,"tag":287,"props":570,"children":571},{"style":294},[572],{"type":45,"value":573},"concurrency",{"type":39,"tag":287,"props":575,"children":576},{"style":300},[577],{"type":45,"value":303},{"type":39,"tag":287,"props":579,"children":580},{"class":289,"line":306},[581,586,590],{"type":39,"tag":287,"props":582,"children":583},{"style":294},[584],{"type":45,"value":585},"  max",{"type":39,"tag":287,"props":587,"children":588},{"style":300},[589],{"type":45,"value":317},{"type":39,"tag":287,"props":591,"children":593},{"style":592},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[594],{"type":45,"value":595}," 32\n",{"type":39,"tag":40,"props":597,"children":599},{"id":598},"predictpy-essentials",[600],{"type":45,"value":601},"predict.py essentials",{"type":39,"tag":255,"props":603,"children":607},{"className":604,"code":605,"language":606,"meta":263,"style":263},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from cog import BasePredictor, Input, Path\n\nclass Predictor(BasePredictor):\n    def setup(self) -> None:\n        \"\"\"One-time loads. Heavy work goes here, not in predict().\"\"\"\n        self.model = load_model(\"weights\u002F\")\n\n    def predict(\n        self,\n        prompt: str = Input(description=\"Text prompt for generation\"),\n        seed: int = Input(description=\"Random seed; leave blank for random\", default=None),\n        num_steps: int = Input(description=\"Number of denoising steps\", ge=1, le=50, default=20),\n        output_format: str = Input(description=\"Output image format\", choices=[\"webp\", \"jpg\", \"png\"], default=\"webp\"),\n    ) -> Path:\n        \"\"\"Run a single prediction.\"\"\"\n        if not prompt.strip():\n            raise ValueError(\"prompt cannot be empty\")\n        out = self.model.generate(prompt, seed=seed, steps=num_steps)\n        return Path(out)\n","python",[608],{"type":39,"tag":69,"props":609,"children":610},{"__ignoreMap":263},[611,619,628,636,644,652,660,667,675,683,692,701,710,719,728,737,746,755,764],{"type":39,"tag":287,"props":612,"children":613},{"class":289,"line":290},[614],{"type":39,"tag":287,"props":615,"children":616},{},[617],{"type":45,"value":618},"from cog import BasePredictor, Input, Path\n",{"type":39,"tag":287,"props":620,"children":621},{"class":289,"line":306},[622],{"type":39,"tag":287,"props":623,"children":625},{"emptyLinePlaceholder":624},true,[626],{"type":45,"value":627},"\n",{"type":39,"tag":287,"props":629,"children":630},{"class":289,"line":326},[631],{"type":39,"tag":287,"props":632,"children":633},{},[634],{"type":45,"value":635},"class Predictor(BasePredictor):\n",{"type":39,"tag":287,"props":637,"children":638},{"class":289,"line":355},[639],{"type":39,"tag":287,"props":640,"children":641},{},[642],{"type":45,"value":643},"    def setup(self) -> None:\n",{"type":39,"tag":287,"props":645,"children":646},{"class":289,"line":381},[647],{"type":39,"tag":287,"props":648,"children":649},{},[650],{"type":45,"value":651},"        \"\"\"One-time loads. Heavy work goes here, not in predict().\"\"\"\n",{"type":39,"tag":287,"props":653,"children":654},{"class":289,"line":26},[655],{"type":39,"tag":287,"props":656,"children":657},{},[658],{"type":45,"value":659},"        self.model = load_model(\"weights\u002F\")\n",{"type":39,"tag":287,"props":661,"children":662},{"class":289,"line":411},[663],{"type":39,"tag":287,"props":664,"children":665},{"emptyLinePlaceholder":624},[666],{"type":45,"value":627},{"type":39,"tag":287,"props":668,"children":669},{"class":289,"line":425},[670],{"type":39,"tag":287,"props":671,"children":672},{},[673],{"type":45,"value":674},"    def predict(\n",{"type":39,"tag":287,"props":676,"children":677},{"class":289,"line":438},[678],{"type":39,"tag":287,"props":679,"children":680},{},[681],{"type":45,"value":682},"        self,\n",{"type":39,"tag":287,"props":684,"children":686},{"class":289,"line":685},10,[687],{"type":39,"tag":287,"props":688,"children":689},{},[690],{"type":45,"value":691},"        prompt: str = Input(description=\"Text prompt for generation\"),\n",{"type":39,"tag":287,"props":693,"children":695},{"class":289,"line":694},11,[696],{"type":39,"tag":287,"props":697,"children":698},{},[699],{"type":45,"value":700},"        seed: int = Input(description=\"Random seed; leave blank for random\", default=None),\n",{"type":39,"tag":287,"props":702,"children":704},{"class":289,"line":703},12,[705],{"type":39,"tag":287,"props":706,"children":707},{},[708],{"type":45,"value":709},"        num_steps: int = Input(description=\"Number of denoising steps\", ge=1, le=50, default=20),\n",{"type":39,"tag":287,"props":711,"children":713},{"class":289,"line":712},13,[714],{"type":39,"tag":287,"props":715,"children":716},{},[717],{"type":45,"value":718},"        output_format: str = Input(description=\"Output image format\", choices=[\"webp\", \"jpg\", \"png\"], default=\"webp\"),\n",{"type":39,"tag":287,"props":720,"children":722},{"class":289,"line":721},14,[723],{"type":39,"tag":287,"props":724,"children":725},{},[726],{"type":45,"value":727},"    ) -> Path:\n",{"type":39,"tag":287,"props":729,"children":731},{"class":289,"line":730},15,[732],{"type":39,"tag":287,"props":733,"children":734},{},[735],{"type":45,"value":736},"        \"\"\"Run a single prediction.\"\"\"\n",{"type":39,"tag":287,"props":738,"children":740},{"class":289,"line":739},16,[741],{"type":39,"tag":287,"props":742,"children":743},{},[744],{"type":45,"value":745},"        if not prompt.strip():\n",{"type":39,"tag":287,"props":747,"children":749},{"class":289,"line":748},17,[750],{"type":39,"tag":287,"props":751,"children":752},{},[753],{"type":45,"value":754},"            raise ValueError(\"prompt cannot be empty\")\n",{"type":39,"tag":287,"props":756,"children":758},{"class":289,"line":757},18,[759],{"type":39,"tag":287,"props":760,"children":761},{},[762],{"type":45,"value":763},"        out = self.model.generate(prompt, seed=seed, steps=num_steps)\n",{"type":39,"tag":287,"props":765,"children":767},{"class":289,"line":766},19,[768],{"type":39,"tag":287,"props":769,"children":770},{},[771],{"type":45,"value":772},"        return Path(out)\n",{"type":39,"tag":249,"props":774,"children":775},{},[776],{"type":45,"value":777},"Input rules:",{"type":39,"tag":48,"props":779,"children":780},{},[781,794,830,842,861,874],{"type":39,"tag":52,"props":782,"children":783},{},[784,786,792],{"type":45,"value":785},"Every input needs a ",{"type":39,"tag":69,"props":787,"children":789},{"className":788},[],[790],{"type":45,"value":791},"description",{"type":45,"value":793},". The description shows up in the model schema and on Replicate's web UI.",{"type":39,"tag":52,"props":795,"children":796},{},[797,798,804,806,812,814,820,822,828],{"type":45,"value":479},{"type":39,"tag":69,"props":799,"children":801},{"className":800},[],[802],{"type":45,"value":803},"ge",{"type":45,"value":805},"\u002F",{"type":39,"tag":69,"props":807,"children":809},{"className":808},[],[810],{"type":45,"value":811},"le",{"type":45,"value":813}," for numeric bounds, ",{"type":39,"tag":69,"props":815,"children":817},{"className":816},[],[818],{"type":45,"value":819},"choices=[...]",{"type":45,"value":821}," for enums, ",{"type":39,"tag":69,"props":823,"children":825},{"className":824},[],[826],{"type":45,"value":827},"regex=",{"type":45,"value":829}," for strings.",{"type":39,"tag":52,"props":831,"children":832},{},[833,834,840],{"type":45,"value":479},{"type":39,"tag":69,"props":835,"children":837},{"className":836},[],[838],{"type":45,"value":839},"cog.Path",{"type":45,"value":841}," for file inputs and outputs, never raw bytes.",{"type":39,"tag":52,"props":843,"children":844},{},[845,846,852,854,860],{"type":45,"value":479},{"type":39,"tag":69,"props":847,"children":849},{"className":848},[],[850],{"type":45,"value":851},"cog.Secret",{"type":45,"value":853}," for any token-like input (HF tokens, API keys), never plain ",{"type":39,"tag":69,"props":855,"children":857},{"className":856},[],[858],{"type":45,"value":859},"str",{"type":45,"value":157},{"type":39,"tag":52,"props":862,"children":863},{},[864,866,872],{"type":45,"value":865},"Provide a default that's inside ",{"type":39,"tag":69,"props":867,"children":869},{"className":868},[],[870],{"type":45,"value":871},"choices",{"type":45,"value":873}," for categorical inputs.",{"type":39,"tag":52,"props":875,"children":876},{},[877,879,885,887,893],{"type":45,"value":878},"Validate inputs early in ",{"type":39,"tag":69,"props":880,"children":882},{"className":881},[],[883],{"type":45,"value":884},"predict()",{"type":45,"value":886}," and raise ",{"type":39,"tag":69,"props":888,"children":890},{"className":889},[],[891],{"type":45,"value":892},"ValueError",{"type":45,"value":157},{"type":39,"tag":249,"props":895,"children":896},{},[897],{"type":45,"value":898},"Streaming text output (for LLMs):",{"type":39,"tag":255,"props":900,"children":902},{"className":604,"code":901,"language":606,"meta":263,"style":263},"from cog import BasePredictor, Input, ConcatenateIterator\n\nclass Predictor(BasePredictor):\n    def predict(self, prompt: str = Input(description=\"Prompt\")) -> ConcatenateIterator[str]:\n        for token in self.model.stream(prompt):\n            yield token\n",[903],{"type":39,"tag":69,"props":904,"children":905},{"__ignoreMap":263},[906,914,921,928,936,944],{"type":39,"tag":287,"props":907,"children":908},{"class":289,"line":290},[909],{"type":39,"tag":287,"props":910,"children":911},{},[912],{"type":45,"value":913},"from cog import BasePredictor, Input, ConcatenateIterator\n",{"type":39,"tag":287,"props":915,"children":916},{"class":289,"line":306},[917],{"type":39,"tag":287,"props":918,"children":919},{"emptyLinePlaceholder":624},[920],{"type":45,"value":627},{"type":39,"tag":287,"props":922,"children":923},{"class":289,"line":326},[924],{"type":39,"tag":287,"props":925,"children":926},{},[927],{"type":45,"value":635},{"type":39,"tag":287,"props":929,"children":930},{"class":289,"line":355},[931],{"type":39,"tag":287,"props":932,"children":933},{},[934],{"type":45,"value":935},"    def predict(self, prompt: str = Input(description=\"Prompt\")) -> ConcatenateIterator[str]:\n",{"type":39,"tag":287,"props":937,"children":938},{"class":289,"line":381},[939],{"type":39,"tag":287,"props":940,"children":941},{},[942],{"type":45,"value":943},"        for token in self.model.stream(prompt):\n",{"type":39,"tag":287,"props":945,"children":946},{"class":289,"line":26},[947],{"type":39,"tag":287,"props":948,"children":949},{},[950],{"type":45,"value":951},"            yield token\n",{"type":39,"tag":249,"props":953,"children":954},{},[955,957,963],{"type":45,"value":956},"Async predictor with continuous batching (paired with ",{"type":39,"tag":69,"props":958,"children":960},{"className":959},[],[961],{"type":45,"value":962},"concurrency.max",{"type":45,"value":964}," in cog.yaml):",{"type":39,"tag":255,"props":966,"children":968},{"className":604,"code":967,"language":606,"meta":263,"style":263},"from cog import BasePredictor, Input, AsyncConcatenateIterator\n\nclass Predictor(BasePredictor):\n    async def setup(self) -> None:\n        self.engine = await load_async_engine()\n\n    async def predict(\n        self,\n        prompt: str = Input(description=\"Prompt\"),\n    ) -> AsyncConcatenateIterator[str]:\n        async for token in self.engine.generate(prompt):\n            yield token\n",[969],{"type":39,"tag":69,"props":970,"children":971},{"__ignoreMap":263},[972,980,987,994,1002,1010,1017,1025,1032,1040,1048,1056],{"type":39,"tag":287,"props":973,"children":974},{"class":289,"line":290},[975],{"type":39,"tag":287,"props":976,"children":977},{},[978],{"type":45,"value":979},"from cog import BasePredictor, Input, AsyncConcatenateIterator\n",{"type":39,"tag":287,"props":981,"children":982},{"class":289,"line":306},[983],{"type":39,"tag":287,"props":984,"children":985},{"emptyLinePlaceholder":624},[986],{"type":45,"value":627},{"type":39,"tag":287,"props":988,"children":989},{"class":289,"line":326},[990],{"type":39,"tag":287,"props":991,"children":992},{},[993],{"type":45,"value":635},{"type":39,"tag":287,"props":995,"children":996},{"class":289,"line":355},[997],{"type":39,"tag":287,"props":998,"children":999},{},[1000],{"type":45,"value":1001},"    async def setup(self) -> None:\n",{"type":39,"tag":287,"props":1003,"children":1004},{"class":289,"line":381},[1005],{"type":39,"tag":287,"props":1006,"children":1007},{},[1008],{"type":45,"value":1009},"        self.engine = await load_async_engine()\n",{"type":39,"tag":287,"props":1011,"children":1012},{"class":289,"line":26},[1013],{"type":39,"tag":287,"props":1014,"children":1015},{"emptyLinePlaceholder":624},[1016],{"type":45,"value":627},{"type":39,"tag":287,"props":1018,"children":1019},{"class":289,"line":411},[1020],{"type":39,"tag":287,"props":1021,"children":1022},{},[1023],{"type":45,"value":1024},"    async def predict(\n",{"type":39,"tag":287,"props":1026,"children":1027},{"class":289,"line":425},[1028],{"type":39,"tag":287,"props":1029,"children":1030},{},[1031],{"type":45,"value":682},{"type":39,"tag":287,"props":1033,"children":1034},{"class":289,"line":438},[1035],{"type":39,"tag":287,"props":1036,"children":1037},{},[1038],{"type":45,"value":1039},"        prompt: str = Input(description=\"Prompt\"),\n",{"type":39,"tag":287,"props":1041,"children":1042},{"class":289,"line":685},[1043],{"type":39,"tag":287,"props":1044,"children":1045},{},[1046],{"type":45,"value":1047},"    ) -> AsyncConcatenateIterator[str]:\n",{"type":39,"tag":287,"props":1049,"children":1050},{"class":289,"line":694},[1051],{"type":39,"tag":287,"props":1052,"children":1053},{},[1054],{"type":45,"value":1055},"        async for token in self.engine.generate(prompt):\n",{"type":39,"tag":287,"props":1057,"children":1058},{"class":289,"line":703},[1059],{"type":39,"tag":287,"props":1060,"children":1061},{},[1062],{"type":45,"value":951},{"type":39,"tag":249,"props":1064,"children":1065},{},[1066,1068,1073,1075,1081],{"type":45,"value":1067},"Dynamic ",{"type":39,"tag":69,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":45,"value":871},{"type":45,"value":1074}," from on-disk assets (e.g. a ",{"type":39,"tag":69,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":45,"value":1080},"voices\u002F",{"type":45,"value":1082}," directory of audio samples):",{"type":39,"tag":255,"props":1084,"children":1086},{"className":604,"code":1085,"language":606,"meta":263,"style":263},"from pathlib import Path as _P\nAVAILABLE_VOICES = sorted(p.stem for p in _P(\"voices\").glob(\"*.wav\"))\n\nclass Predictor(BasePredictor):\n    def predict(\n        self,\n        speaker: str = Input(description=\"Voice\", choices=AVAILABLE_VOICES, default=AVAILABLE_VOICES[0]),\n    ) -> Path: ...\n",[1087],{"type":39,"tag":69,"props":1088,"children":1089},{"__ignoreMap":263},[1090,1098,1106,1113,1120,1127,1134,1142],{"type":39,"tag":287,"props":1091,"children":1092},{"class":289,"line":290},[1093],{"type":39,"tag":287,"props":1094,"children":1095},{},[1096],{"type":45,"value":1097},"from pathlib import Path as _P\n",{"type":39,"tag":287,"props":1099,"children":1100},{"class":289,"line":306},[1101],{"type":39,"tag":287,"props":1102,"children":1103},{},[1104],{"type":45,"value":1105},"AVAILABLE_VOICES = sorted(p.stem for p in _P(\"voices\").glob(\"*.wav\"))\n",{"type":39,"tag":287,"props":1107,"children":1108},{"class":289,"line":326},[1109],{"type":39,"tag":287,"props":1110,"children":1111},{"emptyLinePlaceholder":624},[1112],{"type":45,"value":627},{"type":39,"tag":287,"props":1114,"children":1115},{"class":289,"line":355},[1116],{"type":39,"tag":287,"props":1117,"children":1118},{},[1119],{"type":45,"value":635},{"type":39,"tag":287,"props":1121,"children":1122},{"class":289,"line":381},[1123],{"type":39,"tag":287,"props":1124,"children":1125},{},[1126],{"type":45,"value":674},{"type":39,"tag":287,"props":1128,"children":1129},{"class":289,"line":26},[1130],{"type":39,"tag":287,"props":1131,"children":1132},{},[1133],{"type":45,"value":682},{"type":39,"tag":287,"props":1135,"children":1136},{"class":289,"line":411},[1137],{"type":39,"tag":287,"props":1138,"children":1139},{},[1140],{"type":45,"value":1141},"        speaker: str = Input(description=\"Voice\", choices=AVAILABLE_VOICES, default=AVAILABLE_VOICES[0]),\n",{"type":39,"tag":287,"props":1143,"children":1144},{"class":289,"line":425},[1145],{"type":39,"tag":287,"props":1146,"children":1147},{},[1148],{"type":45,"value":1149},"    ) -> Path: ...\n",{"type":39,"tag":40,"props":1151,"children":1153},{"id":1152},"loading-weights-fast",[1154],{"type":45,"value":1155},"Loading weights fast",{"type":39,"tag":249,"props":1157,"children":1158},{},[1159],{"type":45,"value":1160},"Cold boot dominates user-perceived latency. Three patterns, ranked by simplicity:",{"type":39,"tag":1162,"props":1163,"children":1165},"h3",{"id":1164},"_1-bake-weights-into-the-image-at-build-time",[1166],{"type":45,"value":1167},"1. Bake weights into the image at build time",{"type":39,"tag":249,"props":1169,"children":1170},{},[1171],{"type":45,"value":1172},"Best for small or medium weights (\u003C 5GB) that you want zero-cold-boot for.",{"type":39,"tag":249,"props":1174,"children":1175},{},[1176],{"type":45,"value":1177},"For torchvision:",{"type":39,"tag":255,"props":1179,"children":1181},{"className":604,"code":1180,"language":606,"meta":263,"style":263},"import os\nos.environ[\"TORCH_HOME\"] = \".\"  # set before importing torch\nimport torch\nfrom torchvision import models\n",[1182],{"type":39,"tag":69,"props":1183,"children":1184},{"__ignoreMap":263},[1185,1193,1201,1209],{"type":39,"tag":287,"props":1186,"children":1187},{"class":289,"line":290},[1188],{"type":39,"tag":287,"props":1189,"children":1190},{},[1191],{"type":45,"value":1192},"import os\n",{"type":39,"tag":287,"props":1194,"children":1195},{"class":289,"line":306},[1196],{"type":39,"tag":287,"props":1197,"children":1198},{},[1199],{"type":45,"value":1200},"os.environ[\"TORCH_HOME\"] = \".\"  # set before importing torch\n",{"type":39,"tag":287,"props":1202,"children":1203},{"class":289,"line":326},[1204],{"type":39,"tag":287,"props":1205,"children":1206},{},[1207],{"type":45,"value":1208},"import torch\n",{"type":39,"tag":287,"props":1210,"children":1211},{"class":289,"line":355},[1212],{"type":39,"tag":287,"props":1213,"children":1214},{},[1215],{"type":45,"value":1216},"from torchvision import models\n",{"type":39,"tag":249,"props":1218,"children":1219},{},[1220],{"type":45,"value":1221},"For HuggingFace:",{"type":39,"tag":255,"props":1223,"children":1225},{"className":604,"code":1224,"language":606,"meta":263,"style":263},"import os\nos.environ[\"HF_HUB_CACHE\"] = \".\u002F.cache\"\nos.environ[\"HF_XET_HIGH_PERFORMANCE\"] = \"1\"\n",[1226],{"type":39,"tag":69,"props":1227,"children":1228},{"__ignoreMap":263},[1229,1236,1244],{"type":39,"tag":287,"props":1230,"children":1231},{"class":289,"line":290},[1232],{"type":39,"tag":287,"props":1233,"children":1234},{},[1235],{"type":45,"value":1192},{"type":39,"tag":287,"props":1237,"children":1238},{"class":289,"line":306},[1239],{"type":39,"tag":287,"props":1240,"children":1241},{},[1242],{"type":45,"value":1243},"os.environ[\"HF_HUB_CACHE\"] = \".\u002F.cache\"\n",{"type":39,"tag":287,"props":1245,"children":1246},{"class":289,"line":326},[1247],{"type":39,"tag":287,"props":1248,"children":1249},{},[1250],{"type":45,"value":1251},"os.environ[\"HF_XET_HIGH_PERFORMANCE\"] = \"1\"\n",{"type":39,"tag":249,"props":1253,"children":1254},{},[1255,1257,1263,1265,1271],{"type":45,"value":1256},"Then download once during ",{"type":39,"tag":69,"props":1258,"children":1260},{"className":1259},[],[1261],{"type":45,"value":1262},"cog build",{"type":45,"value":1264}," (e.g. in a ",{"type":39,"tag":69,"props":1266,"children":1268},{"className":1267},[],[1269],{"type":45,"value":1270},"run:",{"type":45,"value":1272}," step or by running a small fetcher script as part of the build). The weights become part of the image layer.",{"type":39,"tag":1162,"props":1274,"children":1276},{"id":1275},"_2-pull-from-weightsreplicatedelivery-with-pget",[1277,1279,1285],{"type":45,"value":1278},"2. Pull from ",{"type":39,"tag":69,"props":1280,"children":1282},{"className":1281},[],[1283],{"type":45,"value":1284},"weights.replicate.delivery",{"type":45,"value":1286}," with pget",{"type":39,"tag":249,"props":1288,"children":1289},{},[1290,1292,1298],{"type":45,"value":1291},"Best for large weights, or when you want to share weights across multiple models. ",{"type":39,"tag":69,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":45,"value":1297},"pget",{"type":45,"value":1299}," is Replicate's parallel HTTP fetcher.",{"type":39,"tag":249,"props":1301,"children":1302},{},[1303,1305,1310],{"type":45,"value":1304},"In ",{"type":39,"tag":69,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":45,"value":74},{"type":45,"value":317},{"type":39,"tag":255,"props":1312,"children":1314},{"className":279,"code":1313,"language":281,"meta":263,"style":263},"build:\n  run:\n    - curl -o \u002Fusr\u002Flocal\u002Fbin\u002Fpget -L \"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fpget\u002Freleases\u002Fdownload\u002Fv0.8.2\u002Fpget_linux_x86_64\"\n    - chmod +x \u002Fusr\u002Flocal\u002Fbin\u002Fpget\n",[1315],{"type":39,"tag":69,"props":1316,"children":1317},{"__ignoreMap":263},[1318,1329,1341,1353],{"type":39,"tag":287,"props":1319,"children":1320},{"class":289,"line":290},[1321,1325],{"type":39,"tag":287,"props":1322,"children":1323},{"style":294},[1324],{"type":45,"value":297},{"type":39,"tag":287,"props":1326,"children":1327},{"style":300},[1328],{"type":45,"value":303},{"type":39,"tag":287,"props":1330,"children":1331},{"class":289,"line":306},[1332,1337],{"type":39,"tag":287,"props":1333,"children":1334},{"style":294},[1335],{"type":45,"value":1336},"  run",{"type":39,"tag":287,"props":1338,"children":1339},{"style":300},[1340],{"type":45,"value":303},{"type":39,"tag":287,"props":1342,"children":1343},{"class":289,"line":326},[1344,1348],{"type":39,"tag":287,"props":1345,"children":1346},{"style":300},[1347],{"type":45,"value":417},{"type":39,"tag":287,"props":1349,"children":1350},{"style":344},[1351],{"type":45,"value":1352}," curl -o \u002Fusr\u002Flocal\u002Fbin\u002Fpget -L \"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fpget\u002Freleases\u002Fdownload\u002Fv0.8.2\u002Fpget_linux_x86_64\"\n",{"type":39,"tag":287,"props":1354,"children":1355},{"class":289,"line":355},[1356,1360],{"type":39,"tag":287,"props":1357,"children":1358},{"style":300},[1359],{"type":45,"value":417},{"type":39,"tag":287,"props":1361,"children":1362},{"style":344},[1363],{"type":45,"value":1364}," chmod +x \u002Fusr\u002Flocal\u002Fbin\u002Fpget\n",{"type":39,"tag":249,"props":1366,"children":1367},{},[1368,1369,1375],{"type":45,"value":1304},{"type":39,"tag":69,"props":1370,"children":1372},{"className":1371},[],[1373],{"type":45,"value":1374},"setup()",{"type":45,"value":317},{"type":39,"tag":255,"props":1377,"children":1379},{"className":604,"code":1378,"language":606,"meta":263,"style":263},"import subprocess\nfrom pathlib import Path\n\nWEIGHTS_URL = \"https:\u002F\u002Fweights.replicate.delivery\u002Fdefault\u002Fmy-model\u002Fweights.tar\"\nWEIGHTS_DIR = Path(\"weights\")\n\nclass Predictor(BasePredictor):\n    def setup(self) -> None:\n        if not WEIGHTS_DIR.exists():\n            # -x extracts tar in-memory; default concurrency is 4 * NumCPU\n            subprocess.check_call([\"pget\", \"-x\", WEIGHTS_URL, str(WEIGHTS_DIR)])\n        self.model = load_from(WEIGHTS_DIR)\n",[1380],{"type":39,"tag":69,"props":1381,"children":1382},{"__ignoreMap":263},[1383,1391,1399,1406,1414,1422,1429,1436,1443,1451,1459,1467],{"type":39,"tag":287,"props":1384,"children":1385},{"class":289,"line":290},[1386],{"type":39,"tag":287,"props":1387,"children":1388},{},[1389],{"type":45,"value":1390},"import subprocess\n",{"type":39,"tag":287,"props":1392,"children":1393},{"class":289,"line":306},[1394],{"type":39,"tag":287,"props":1395,"children":1396},{},[1397],{"type":45,"value":1398},"from pathlib import Path\n",{"type":39,"tag":287,"props":1400,"children":1401},{"class":289,"line":326},[1402],{"type":39,"tag":287,"props":1403,"children":1404},{"emptyLinePlaceholder":624},[1405],{"type":45,"value":627},{"type":39,"tag":287,"props":1407,"children":1408},{"class":289,"line":355},[1409],{"type":39,"tag":287,"props":1410,"children":1411},{},[1412],{"type":45,"value":1413},"WEIGHTS_URL = \"https:\u002F\u002Fweights.replicate.delivery\u002Fdefault\u002Fmy-model\u002Fweights.tar\"\n",{"type":39,"tag":287,"props":1415,"children":1416},{"class":289,"line":381},[1417],{"type":39,"tag":287,"props":1418,"children":1419},{},[1420],{"type":45,"value":1421},"WEIGHTS_DIR = Path(\"weights\")\n",{"type":39,"tag":287,"props":1423,"children":1424},{"class":289,"line":26},[1425],{"type":39,"tag":287,"props":1426,"children":1427},{"emptyLinePlaceholder":624},[1428],{"type":45,"value":627},{"type":39,"tag":287,"props":1430,"children":1431},{"class":289,"line":411},[1432],{"type":39,"tag":287,"props":1433,"children":1434},{},[1435],{"type":45,"value":635},{"type":39,"tag":287,"props":1437,"children":1438},{"class":289,"line":425},[1439],{"type":39,"tag":287,"props":1440,"children":1441},{},[1442],{"type":45,"value":643},{"type":39,"tag":287,"props":1444,"children":1445},{"class":289,"line":438},[1446],{"type":39,"tag":287,"props":1447,"children":1448},{},[1449],{"type":45,"value":1450},"        if not WEIGHTS_DIR.exists():\n",{"type":39,"tag":287,"props":1452,"children":1453},{"class":289,"line":685},[1454],{"type":39,"tag":287,"props":1455,"children":1456},{},[1457],{"type":45,"value":1458},"            # -x extracts tar in-memory; default concurrency is 4 * NumCPU\n",{"type":39,"tag":287,"props":1460,"children":1461},{"class":289,"line":694},[1462],{"type":39,"tag":287,"props":1463,"children":1464},{},[1465],{"type":45,"value":1466},"            subprocess.check_call([\"pget\", \"-x\", WEIGHTS_URL, str(WEIGHTS_DIR)])\n",{"type":39,"tag":287,"props":1468,"children":1469},{"class":289,"line":703},[1470],{"type":39,"tag":287,"props":1471,"children":1472},{},[1473],{"type":45,"value":1474},"        self.model = load_from(WEIGHTS_DIR)\n",{"type":39,"tag":249,"props":1476,"children":1477},{},[1478],{"type":45,"value":1479},"For multiple files in one shot:",{"type":39,"tag":255,"props":1481,"children":1483},{"className":604,"code":1482,"language":606,"meta":263,"style":263},"manifest = \"\\n\".join([\n    f\"{base}\u002Funet.safetensors weights\u002Funet.safetensors\",\n    f\"{base}\u002Fvae.safetensors  weights\u002Fvae.safetensors\",\n    f\"{base}\u002Ftext_encoder.safetensors weights\u002Ftext_encoder.safetensors\",\n])\nsubprocess.run([\"pget\", \"multifile\", \"-\"], input=manifest, text=True, check=True)\n",[1484],{"type":39,"tag":69,"props":1485,"children":1486},{"__ignoreMap":263},[1487,1495,1503,1511,1519,1527],{"type":39,"tag":287,"props":1488,"children":1489},{"class":289,"line":290},[1490],{"type":39,"tag":287,"props":1491,"children":1492},{},[1493],{"type":45,"value":1494},"manifest = \"\\n\".join([\n",{"type":39,"tag":287,"props":1496,"children":1497},{"class":289,"line":306},[1498],{"type":39,"tag":287,"props":1499,"children":1500},{},[1501],{"type":45,"value":1502},"    f\"{base}\u002Funet.safetensors weights\u002Funet.safetensors\",\n",{"type":39,"tag":287,"props":1504,"children":1505},{"class":289,"line":326},[1506],{"type":39,"tag":287,"props":1507,"children":1508},{},[1509],{"type":45,"value":1510},"    f\"{base}\u002Fvae.safetensors  weights\u002Fvae.safetensors\",\n",{"type":39,"tag":287,"props":1512,"children":1513},{"class":289,"line":355},[1514],{"type":39,"tag":287,"props":1515,"children":1516},{},[1517],{"type":45,"value":1518},"    f\"{base}\u002Ftext_encoder.safetensors weights\u002Ftext_encoder.safetensors\",\n",{"type":39,"tag":287,"props":1520,"children":1521},{"class":289,"line":381},[1522],{"type":39,"tag":287,"props":1523,"children":1524},{},[1525],{"type":45,"value":1526},"])\n",{"type":39,"tag":287,"props":1528,"children":1529},{"class":289,"line":26},[1530],{"type":39,"tag":287,"props":1531,"children":1532},{},[1533],{"type":45,"value":1534},"subprocess.run([\"pget\", \"multifile\", \"-\"], input=manifest, text=True, check=True)\n",{"type":39,"tag":1162,"props":1536,"children":1538},{"id":1537},"_3-huggingface-hub-with-hf_transfer",[1539],{"type":45,"value":1540},"3. HuggingFace Hub with hf_transfer",{"type":39,"tag":249,"props":1542,"children":1543},{},[1544,1546,1552,1554,1560,1561,1567,1569,1574],{"type":45,"value":1545},"Set ",{"type":39,"tag":69,"props":1547,"children":1549},{"className":1548},[],[1550],{"type":45,"value":1551},"HF_HUB_ENABLE_HF_TRANSFER=1",{"type":45,"value":1553}," and use ",{"type":39,"tag":69,"props":1555,"children":1557},{"className":1556},[],[1558],{"type":45,"value":1559},"huggingface_hub.snapshot_download",{"type":45,"value":208},{"type":39,"tag":69,"props":1562,"children":1564},{"className":1563},[],[1565],{"type":45,"value":1566},"from_pretrained",{"type":45,"value":1568},". Faster than vanilla HF downloads. Use a ",{"type":39,"tag":69,"props":1570,"children":1572},{"className":1571},[],[1573],{"type":45,"value":851},{"type":45,"value":1575}," input for gated models.",{"type":39,"tag":40,"props":1577,"children":1579},{"id":1578},"weight-cache-for-user-supplied-weights",[1580],{"type":45,"value":1581},"Weight cache for user-supplied weights",{"type":39,"tag":249,"props":1583,"children":1584},{},[1585],{"type":45,"value":1586},"For LoRAs or any weights URL the user passes at predict time, use a sha256-keyed disk cache with LRU eviction:",{"type":39,"tag":255,"props":1588,"children":1590},{"className":604,"code":1589,"language":606,"meta":263,"style":263},"import hashlib, shutil, subprocess\nfrom pathlib import Path\n\nclass WeightsDownloadCache:\n    def __init__(self, cache_dir: str = \"\u002Ftmp\u002Fweights-cache\", min_disk_free_gb: int = 10):\n        self.cache_dir = Path(cache_dir)\n        self.cache_dir.mkdir(parents=True, exist_ok=True)\n        self.min_disk_free = min_disk_free_gb * 1024**3\n\n    def ensure(self, url: str) -> Path:\n        key = hashlib.sha256(url.encode()).hexdigest()\n        target = self.cache_dir \u002F key\n        if target.exists():\n            target.touch()  # bump LRU mtime\n            return target\n        self._evict_until_room()\n        subprocess.check_call([\"pget\", url, str(target)])\n        return target\n\n    def _evict_until_room(self) -> None:\n        while shutil.disk_usage(self.cache_dir).free \u003C self.min_disk_free:\n            entries = sorted(self.cache_dir.iterdir(), key=lambda p: p.stat().st_mtime)\n            if not entries:\n                return\n            entries[0].unlink()\n",[1591],{"type":39,"tag":69,"props":1592,"children":1593},{"__ignoreMap":263},[1594,1602,1609,1616,1624,1632,1640,1648,1656,1663,1671,1679,1687,1695,1703,1711,1719,1727,1735,1742,1751,1760,1769,1778,1787],{"type":39,"tag":287,"props":1595,"children":1596},{"class":289,"line":290},[1597],{"type":39,"tag":287,"props":1598,"children":1599},{},[1600],{"type":45,"value":1601},"import hashlib, shutil, subprocess\n",{"type":39,"tag":287,"props":1603,"children":1604},{"class":289,"line":306},[1605],{"type":39,"tag":287,"props":1606,"children":1607},{},[1608],{"type":45,"value":1398},{"type":39,"tag":287,"props":1610,"children":1611},{"class":289,"line":326},[1612],{"type":39,"tag":287,"props":1613,"children":1614},{"emptyLinePlaceholder":624},[1615],{"type":45,"value":627},{"type":39,"tag":287,"props":1617,"children":1618},{"class":289,"line":355},[1619],{"type":39,"tag":287,"props":1620,"children":1621},{},[1622],{"type":45,"value":1623},"class WeightsDownloadCache:\n",{"type":39,"tag":287,"props":1625,"children":1626},{"class":289,"line":381},[1627],{"type":39,"tag":287,"props":1628,"children":1629},{},[1630],{"type":45,"value":1631},"    def __init__(self, cache_dir: str = \"\u002Ftmp\u002Fweights-cache\", min_disk_free_gb: int = 10):\n",{"type":39,"tag":287,"props":1633,"children":1634},{"class":289,"line":26},[1635],{"type":39,"tag":287,"props":1636,"children":1637},{},[1638],{"type":45,"value":1639},"        self.cache_dir = Path(cache_dir)\n",{"type":39,"tag":287,"props":1641,"children":1642},{"class":289,"line":411},[1643],{"type":39,"tag":287,"props":1644,"children":1645},{},[1646],{"type":45,"value":1647},"        self.cache_dir.mkdir(parents=True, exist_ok=True)\n",{"type":39,"tag":287,"props":1649,"children":1650},{"class":289,"line":425},[1651],{"type":39,"tag":287,"props":1652,"children":1653},{},[1654],{"type":45,"value":1655},"        self.min_disk_free = min_disk_free_gb * 1024**3\n",{"type":39,"tag":287,"props":1657,"children":1658},{"class":289,"line":438},[1659],{"type":39,"tag":287,"props":1660,"children":1661},{"emptyLinePlaceholder":624},[1662],{"type":45,"value":627},{"type":39,"tag":287,"props":1664,"children":1665},{"class":289,"line":685},[1666],{"type":39,"tag":287,"props":1667,"children":1668},{},[1669],{"type":45,"value":1670},"    def ensure(self, url: str) -> Path:\n",{"type":39,"tag":287,"props":1672,"children":1673},{"class":289,"line":694},[1674],{"type":39,"tag":287,"props":1675,"children":1676},{},[1677],{"type":45,"value":1678},"        key = hashlib.sha256(url.encode()).hexdigest()\n",{"type":39,"tag":287,"props":1680,"children":1681},{"class":289,"line":703},[1682],{"type":39,"tag":287,"props":1683,"children":1684},{},[1685],{"type":45,"value":1686},"        target = self.cache_dir \u002F key\n",{"type":39,"tag":287,"props":1688,"children":1689},{"class":289,"line":712},[1690],{"type":39,"tag":287,"props":1691,"children":1692},{},[1693],{"type":45,"value":1694},"        if target.exists():\n",{"type":39,"tag":287,"props":1696,"children":1697},{"class":289,"line":721},[1698],{"type":39,"tag":287,"props":1699,"children":1700},{},[1701],{"type":45,"value":1702},"            target.touch()  # bump LRU mtime\n",{"type":39,"tag":287,"props":1704,"children":1705},{"class":289,"line":730},[1706],{"type":39,"tag":287,"props":1707,"children":1708},{},[1709],{"type":45,"value":1710},"            return target\n",{"type":39,"tag":287,"props":1712,"children":1713},{"class":289,"line":739},[1714],{"type":39,"tag":287,"props":1715,"children":1716},{},[1717],{"type":45,"value":1718},"        self._evict_until_room()\n",{"type":39,"tag":287,"props":1720,"children":1721},{"class":289,"line":748},[1722],{"type":39,"tag":287,"props":1723,"children":1724},{},[1725],{"type":45,"value":1726},"        subprocess.check_call([\"pget\", url, str(target)])\n",{"type":39,"tag":287,"props":1728,"children":1729},{"class":289,"line":757},[1730],{"type":39,"tag":287,"props":1731,"children":1732},{},[1733],{"type":45,"value":1734},"        return target\n",{"type":39,"tag":287,"props":1736,"children":1737},{"class":289,"line":766},[1738],{"type":39,"tag":287,"props":1739,"children":1740},{"emptyLinePlaceholder":624},[1741],{"type":45,"value":627},{"type":39,"tag":287,"props":1743,"children":1745},{"class":289,"line":1744},20,[1746],{"type":39,"tag":287,"props":1747,"children":1748},{},[1749],{"type":45,"value":1750},"    def _evict_until_room(self) -> None:\n",{"type":39,"tag":287,"props":1752,"children":1754},{"class":289,"line":1753},21,[1755],{"type":39,"tag":287,"props":1756,"children":1757},{},[1758],{"type":45,"value":1759},"        while shutil.disk_usage(self.cache_dir).free \u003C self.min_disk_free:\n",{"type":39,"tag":287,"props":1761,"children":1763},{"class":289,"line":1762},22,[1764],{"type":39,"tag":287,"props":1765,"children":1766},{},[1767],{"type":45,"value":1768},"            entries = sorted(self.cache_dir.iterdir(), key=lambda p: p.stat().st_mtime)\n",{"type":39,"tag":287,"props":1770,"children":1772},{"class":289,"line":1771},23,[1773],{"type":39,"tag":287,"props":1774,"children":1775},{},[1776],{"type":45,"value":1777},"            if not entries:\n",{"type":39,"tag":287,"props":1779,"children":1781},{"class":289,"line":1780},24,[1782],{"type":39,"tag":287,"props":1783,"children":1784},{},[1785],{"type":45,"value":1786},"                return\n",{"type":39,"tag":287,"props":1788,"children":1790},{"class":289,"line":1789},25,[1791],{"type":39,"tag":287,"props":1792,"children":1793},{},[1794],{"type":45,"value":1795},"            entries[0].unlink()\n",{"type":39,"tag":249,"props":1797,"children":1798},{},[1799,1801,1807,1809,1815],{"type":45,"value":1800},"See ",{"type":39,"tag":69,"props":1802,"children":1804},{"className":1803},[],[1805],{"type":45,"value":1806},"replicate\u002Fcog-flux\u002Fweights.py",{"type":45,"value":1808}," for a production version that handles HF, CivitAI, Replicate, and arbitrary ",{"type":39,"tag":69,"props":1810,"children":1812},{"className":1811},[],[1813],{"type":45,"value":1814},".safetensors",{"type":45,"value":1816}," URLs.",{"type":39,"tag":40,"props":1818,"children":1820},{"id":1819},"multi-lora-composition",[1821],{"type":45,"value":1822},"Multi-LoRA composition",{"type":39,"tag":249,"props":1824,"children":1825},{},[1826],{"type":45,"value":1827},"Reload only when the URL changes; compose two LoRAs with separate scales:",{"type":39,"tag":255,"props":1829,"children":1831},{"className":604,"code":1830,"language":606,"meta":263,"style":263},"class Predictor(BasePredictor):\n    def setup(self) -> None:\n        self.pipe = load_base_pipeline()\n        self.loaded = {\"main\": None, \"extra\": None}\n\n    def _ensure_lora(self, slot: str, url: str | None) -> None:\n        if url == self.loaded[slot]:\n            return\n        if self.loaded[slot] is not None:\n            self.pipe.unload_lora_weights(adapter_name=slot)\n        if url:\n            path = self.cache.ensure(url)\n            self.pipe.load_lora_weights(str(path), adapter_name=slot)\n        self.loaded[slot] = url\n\n    def predict(\n        self,\n        prompt: str = Input(description=\"Prompt\"),\n        lora_url: str = Input(description=\"Primary LoRA URL\", default=None),\n        lora_scale: float = Input(description=\"Primary LoRA scale\", ge=0.0, le=2.0, default=1.0),\n        extra_lora_url: str = Input(description=\"Optional second LoRA URL\", default=None),\n        extra_lora_scale: float = Input(description=\"Second LoRA scale\", ge=0.0, le=2.0, default=1.0),\n    ) -> Path:\n        self._ensure_lora(\"main\", lora_url)\n        self._ensure_lora(\"extra\", extra_lora_url)\n        adapters = [s for s, u in self.loaded.items() if u]\n        scales = [lora_scale if s == \"main\" else extra_lora_scale for s in adapters]\n        if adapters:\n            self.pipe.set_adapters(adapters, adapter_weights=scales)\n        return Path(self.pipe(prompt).images[0].save(\"\u002Ftmp\u002Fout.png\"))\n",[1832],{"type":39,"tag":69,"props":1833,"children":1834},{"__ignoreMap":263},[1835,1842,1849,1857,1865,1872,1880,1888,1896,1904,1912,1920,1928,1936,1944,1951,1958,1965,1972,1980,1988,1996,2004,2011,2019,2027,2036,2045,2054,2063],{"type":39,"tag":287,"props":1836,"children":1837},{"class":289,"line":290},[1838],{"type":39,"tag":287,"props":1839,"children":1840},{},[1841],{"type":45,"value":635},{"type":39,"tag":287,"props":1843,"children":1844},{"class":289,"line":306},[1845],{"type":39,"tag":287,"props":1846,"children":1847},{},[1848],{"type":45,"value":643},{"type":39,"tag":287,"props":1850,"children":1851},{"class":289,"line":326},[1852],{"type":39,"tag":287,"props":1853,"children":1854},{},[1855],{"type":45,"value":1856},"        self.pipe = load_base_pipeline()\n",{"type":39,"tag":287,"props":1858,"children":1859},{"class":289,"line":355},[1860],{"type":39,"tag":287,"props":1861,"children":1862},{},[1863],{"type":45,"value":1864},"        self.loaded = {\"main\": None, \"extra\": None}\n",{"type":39,"tag":287,"props":1866,"children":1867},{"class":289,"line":381},[1868],{"type":39,"tag":287,"props":1869,"children":1870},{"emptyLinePlaceholder":624},[1871],{"type":45,"value":627},{"type":39,"tag":287,"props":1873,"children":1874},{"class":289,"line":26},[1875],{"type":39,"tag":287,"props":1876,"children":1877},{},[1878],{"type":45,"value":1879},"    def _ensure_lora(self, slot: str, url: str | None) -> None:\n",{"type":39,"tag":287,"props":1881,"children":1882},{"class":289,"line":411},[1883],{"type":39,"tag":287,"props":1884,"children":1885},{},[1886],{"type":45,"value":1887},"        if url == self.loaded[slot]:\n",{"type":39,"tag":287,"props":1889,"children":1890},{"class":289,"line":425},[1891],{"type":39,"tag":287,"props":1892,"children":1893},{},[1894],{"type":45,"value":1895},"            return\n",{"type":39,"tag":287,"props":1897,"children":1898},{"class":289,"line":438},[1899],{"type":39,"tag":287,"props":1900,"children":1901},{},[1902],{"type":45,"value":1903},"        if self.loaded[slot] is not None:\n",{"type":39,"tag":287,"props":1905,"children":1906},{"class":289,"line":685},[1907],{"type":39,"tag":287,"props":1908,"children":1909},{},[1910],{"type":45,"value":1911},"            self.pipe.unload_lora_weights(adapter_name=slot)\n",{"type":39,"tag":287,"props":1913,"children":1914},{"class":289,"line":694},[1915],{"type":39,"tag":287,"props":1916,"children":1917},{},[1918],{"type":45,"value":1919},"        if url:\n",{"type":39,"tag":287,"props":1921,"children":1922},{"class":289,"line":703},[1923],{"type":39,"tag":287,"props":1924,"children":1925},{},[1926],{"type":45,"value":1927},"            path = self.cache.ensure(url)\n",{"type":39,"tag":287,"props":1929,"children":1930},{"class":289,"line":712},[1931],{"type":39,"tag":287,"props":1932,"children":1933},{},[1934],{"type":45,"value":1935},"            self.pipe.load_lora_weights(str(path), adapter_name=slot)\n",{"type":39,"tag":287,"props":1937,"children":1938},{"class":289,"line":721},[1939],{"type":39,"tag":287,"props":1940,"children":1941},{},[1942],{"type":45,"value":1943},"        self.loaded[slot] = url\n",{"type":39,"tag":287,"props":1945,"children":1946},{"class":289,"line":730},[1947],{"type":39,"tag":287,"props":1948,"children":1949},{"emptyLinePlaceholder":624},[1950],{"type":45,"value":627},{"type":39,"tag":287,"props":1952,"children":1953},{"class":289,"line":739},[1954],{"type":39,"tag":287,"props":1955,"children":1956},{},[1957],{"type":45,"value":674},{"type":39,"tag":287,"props":1959,"children":1960},{"class":289,"line":748},[1961],{"type":39,"tag":287,"props":1962,"children":1963},{},[1964],{"type":45,"value":682},{"type":39,"tag":287,"props":1966,"children":1967},{"class":289,"line":757},[1968],{"type":39,"tag":287,"props":1969,"children":1970},{},[1971],{"type":45,"value":1039},{"type":39,"tag":287,"props":1973,"children":1974},{"class":289,"line":766},[1975],{"type":39,"tag":287,"props":1976,"children":1977},{},[1978],{"type":45,"value":1979},"        lora_url: str = Input(description=\"Primary LoRA URL\", default=None),\n",{"type":39,"tag":287,"props":1981,"children":1982},{"class":289,"line":1744},[1983],{"type":39,"tag":287,"props":1984,"children":1985},{},[1986],{"type":45,"value":1987},"        lora_scale: float = Input(description=\"Primary LoRA scale\", ge=0.0, le=2.0, default=1.0),\n",{"type":39,"tag":287,"props":1989,"children":1990},{"class":289,"line":1753},[1991],{"type":39,"tag":287,"props":1992,"children":1993},{},[1994],{"type":45,"value":1995},"        extra_lora_url: str = Input(description=\"Optional second LoRA URL\", default=None),\n",{"type":39,"tag":287,"props":1997,"children":1998},{"class":289,"line":1762},[1999],{"type":39,"tag":287,"props":2000,"children":2001},{},[2002],{"type":45,"value":2003},"        extra_lora_scale: float = Input(description=\"Second LoRA scale\", ge=0.0, le=2.0, default=1.0),\n",{"type":39,"tag":287,"props":2005,"children":2006},{"class":289,"line":1771},[2007],{"type":39,"tag":287,"props":2008,"children":2009},{},[2010],{"type":45,"value":727},{"type":39,"tag":287,"props":2012,"children":2013},{"class":289,"line":1780},[2014],{"type":39,"tag":287,"props":2015,"children":2016},{},[2017],{"type":45,"value":2018},"        self._ensure_lora(\"main\", lora_url)\n",{"type":39,"tag":287,"props":2020,"children":2021},{"class":289,"line":1789},[2022],{"type":39,"tag":287,"props":2023,"children":2024},{},[2025],{"type":45,"value":2026},"        self._ensure_lora(\"extra\", extra_lora_url)\n",{"type":39,"tag":287,"props":2028,"children":2030},{"class":289,"line":2029},26,[2031],{"type":39,"tag":287,"props":2032,"children":2033},{},[2034],{"type":45,"value":2035},"        adapters = [s for s, u in self.loaded.items() if u]\n",{"type":39,"tag":287,"props":2037,"children":2039},{"class":289,"line":2038},27,[2040],{"type":39,"tag":287,"props":2041,"children":2042},{},[2043],{"type":45,"value":2044},"        scales = [lora_scale if s == \"main\" else extra_lora_scale for s in adapters]\n",{"type":39,"tag":287,"props":2046,"children":2048},{"class":289,"line":2047},28,[2049],{"type":39,"tag":287,"props":2050,"children":2051},{},[2052],{"type":45,"value":2053},"        if adapters:\n",{"type":39,"tag":287,"props":2055,"children":2057},{"class":289,"line":2056},29,[2058],{"type":39,"tag":287,"props":2059,"children":2060},{},[2061],{"type":45,"value":2062},"            self.pipe.set_adapters(adapters, adapter_weights=scales)\n",{"type":39,"tag":287,"props":2064,"children":2066},{"class":289,"line":2065},30,[2067],{"type":39,"tag":287,"props":2068,"children":2069},{},[2070],{"type":45,"value":2071},"        return Path(self.pipe(prompt).images[0].save(\"\u002Ftmp\u002Fout.png\"))\n",{"type":39,"tag":40,"props":2073,"children":2075},{"id":2074},"cold-boot-tricks",[2076],{"type":45,"value":2077},"Cold-boot tricks",{"type":39,"tag":249,"props":2079,"children":2080},{},[2081,2083,2089,2090,2096],{"type":45,"value":2082},"From production diffusion models like ",{"type":39,"tag":69,"props":2084,"children":2086},{"className":2085},[],[2087],{"type":45,"value":2088},"replicate\u002Fcog-flux",{"type":45,"value":235},{"type":39,"tag":69,"props":2091,"children":2093},{"className":2092},[],[2094],{"type":45,"value":2095},"replicate\u002Fcog-flux-kontext",{"type":45,"value":317},{"type":39,"tag":48,"props":2098,"children":2099},{},[2100,2149,2177,2229,2234],{"type":39,"tag":52,"props":2101,"children":2102},{},[2103,2105,2110,2111],{"type":45,"value":2104},"Set perf flags once in ",{"type":39,"tag":69,"props":2106,"children":2108},{"className":2107},[],[2109],{"type":45,"value":1374},{"type":45,"value":303},{"type":39,"tag":255,"props":2112,"children":2114},{"className":604,"code":2113,"language":606,"meta":263,"style":263},"import torch\ntorch.set_float32_matmul_precision(\"high\")\ntorch.backends.cuda.matmul.allow_tf32 = True\ntorch.backends.cudnn.benchmark = True\n",[2115],{"type":39,"tag":69,"props":2116,"children":2117},{"__ignoreMap":263},[2118,2125,2133,2141],{"type":39,"tag":287,"props":2119,"children":2120},{"class":289,"line":290},[2121],{"type":39,"tag":287,"props":2122,"children":2123},{},[2124],{"type":45,"value":1208},{"type":39,"tag":287,"props":2126,"children":2127},{"class":289,"line":306},[2128],{"type":39,"tag":287,"props":2129,"children":2130},{},[2131],{"type":45,"value":2132},"torch.set_float32_matmul_precision(\"high\")\n",{"type":39,"tag":287,"props":2134,"children":2135},{"class":289,"line":326},[2136],{"type":39,"tag":287,"props":2137,"children":2138},{},[2139],{"type":45,"value":2140},"torch.backends.cuda.matmul.allow_tf32 = True\n",{"type":39,"tag":287,"props":2142,"children":2143},{"class":289,"line":355},[2144],{"type":39,"tag":287,"props":2145,"children":2146},{},[2147],{"type":45,"value":2148},"torch.backends.cudnn.benchmark = True\n",{"type":39,"tag":52,"props":2150,"children":2151},{},[2152,2154],{"type":45,"value":2153},"Compile and warm up:\n",{"type":39,"tag":255,"props":2155,"children":2157},{"className":604,"code":2156,"language":606,"meta":263,"style":263},"self.model = torch.compile(self.model, dynamic=True)\n_ = self.predict(prompt=\"warmup\", num_steps=1)  # absorbs compile cost in setup\n",[2158],{"type":39,"tag":69,"props":2159,"children":2160},{"__ignoreMap":263},[2161,2169],{"type":39,"tag":287,"props":2162,"children":2163},{"class":289,"line":290},[2164],{"type":39,"tag":287,"props":2165,"children":2166},{},[2167],{"type":45,"value":2168},"self.model = torch.compile(self.model, dynamic=True)\n",{"type":39,"tag":287,"props":2170,"children":2171},{"class":289,"line":306},[2172],{"type":39,"tag":287,"props":2173,"children":2174},{},[2175],{"type":45,"value":2176},"_ = self.predict(prompt=\"warmup\", num_steps=1)  # absorbs compile cost in setup\n",{"type":39,"tag":52,"props":2178,"children":2179},{},[2180,2182,2188,2190],{"type":45,"value":2181},"Load big weights with meta device + ",{"type":39,"tag":69,"props":2183,"children":2185},{"className":2184},[],[2186],{"type":45,"value":2187},"assign=True",{"type":45,"value":2189}," to avoid double-allocating:\n",{"type":39,"tag":255,"props":2191,"children":2193},{"className":604,"code":2192,"language":606,"meta":263,"style":263},"with torch.device(\"meta\"):\n    model = build_model_skeleton()\nstate = torch.load(\"weights.pt\", map_location=\"cpu\")\nmodel.load_state_dict(state, assign=True)\n",[2194],{"type":39,"tag":69,"props":2195,"children":2196},{"__ignoreMap":263},[2197,2205,2213,2221],{"type":39,"tag":287,"props":2198,"children":2199},{"class":289,"line":290},[2200],{"type":39,"tag":287,"props":2201,"children":2202},{},[2203],{"type":45,"value":2204},"with torch.device(\"meta\"):\n",{"type":39,"tag":287,"props":2206,"children":2207},{"class":289,"line":306},[2208],{"type":39,"tag":287,"props":2209,"children":2210},{},[2211],{"type":45,"value":2212},"    model = build_model_skeleton()\n",{"type":39,"tag":287,"props":2214,"children":2215},{"class":289,"line":326},[2216],{"type":39,"tag":287,"props":2217,"children":2218},{},[2219],{"type":45,"value":2220},"state = torch.load(\"weights.pt\", map_location=\"cpu\")\n",{"type":39,"tag":287,"props":2222,"children":2223},{"class":289,"line":355},[2224],{"type":39,"tag":287,"props":2225,"children":2226},{},[2227],{"type":45,"value":2228},"model.load_state_dict(state, assign=True)\n",{"type":39,"tag":52,"props":2230,"children":2231},{},[2232],{"type":45,"value":2233},"Share VAE \u002F text encoder across multiple pipelines (e.g. base + img2img + inpaint) instead of loading three copies.",{"type":39,"tag":52,"props":2235,"children":2236},{},[2237],{"type":45,"value":2238},"For fp8\u002Fint8, save quantized weights ahead of time and load directly; don't quantize at boot.",{"type":39,"tag":40,"props":2240,"children":2242},{"id":2241},"local-development",[2243],{"type":45,"value":2244},"Local development",{"type":39,"tag":255,"props":2246,"children":2249},{"className":2247,"code":2248,"language":45},[258],"cog init                                    # scaffold cog.yaml + predict.py\ncog predict -i prompt=\"hello\"               # build + run a single prediction\ncog predict -i image=@input.jpg -o out.png  # file inputs and outputs\ncog serve -p 8393                           # HTTP server matching production\ncog exec python                             # interactive shell inside the build env\n",[2250],{"type":39,"tag":69,"props":2251,"children":2252},{"__ignoreMap":263},[2253],{"type":45,"value":2248},{"type":39,"tag":40,"props":2255,"children":2257},{"id":2256},"building",[2258],{"type":45,"value":2259},"Building",{"type":39,"tag":255,"props":2261,"children":2264},{"className":2262,"code":2263,"language":45},[258],"cog build -t my-model\ncog build --separate-weights -t my-model    # weights in their own image layer\ncog build --secret id=hf,src=$HOME\u002F.hf_token -t my-model\n",[2265],{"type":39,"tag":69,"props":2266,"children":2267},{"__ignoreMap":263},[2268],{"type":45,"value":2263},{"type":39,"tag":249,"props":2270,"children":2271},{},[2272],{"type":45,"value":2273},"Tips:",{"type":39,"tag":48,"props":2275,"children":2276},{},[2277,2289,2308,2328],{"type":39,"tag":52,"props":2278,"children":2279},{},[2280,2281,2287],{"type":45,"value":479},{"type":39,"tag":69,"props":2282,"children":2284},{"className":2283},[],[2285],{"type":45,"value":2286},"--separate-weights",{"type":45,"value":2288}," for any model with weights > ~1GB. It speeds up cold boots and registry pushes.",{"type":39,"tag":52,"props":2290,"children":2291},{},[2292,2293,2299,2301,2306],{"type":45,"value":479},{"type":39,"tag":69,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":45,"value":2298},"--mount=type=cache,target=\u002Froot\u002F.cache\u002Fpip",{"type":45,"value":2300}," in ",{"type":39,"tag":69,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":45,"value":1270},{"type":45,"value":2307}," steps to cache pip across builds.",{"type":39,"tag":52,"props":2309,"children":2310},{},[2311,2312,2318,2320,2326],{"type":45,"value":479},{"type":39,"tag":69,"props":2313,"children":2315},{"className":2314},[],[2316],{"type":45,"value":2317},"--secret",{"type":45,"value":2319}," instead of ",{"type":39,"tag":69,"props":2321,"children":2323},{"className":2322},[],[2324],{"type":45,"value":2325},"ARG",{"type":45,"value":2327}," to keep tokens out of image history.",{"type":39,"tag":52,"props":2329,"children":2330},{},[2331,2333,2339],{"type":45,"value":2332},"The default Cog base image (",{"type":39,"tag":69,"props":2334,"children":2336},{"className":2335},[],[2337],{"type":45,"value":2338},"--use-cog-base-image=true",{"type":45,"value":2340},") is faster than rolling your own.",{"type":39,"tag":40,"props":2342,"children":2344},{"id":2343},"training",[2345],{"type":45,"value":2346},"Training",{"type":39,"tag":249,"props":2348,"children":2349},{},[2350,2352,2357,2359,2364,2366,2372,2374,2380,2382,2388,2390,2396,2398,2404,2405,2411],{"type":45,"value":2351},"If your model supports fine-tuning, add ",{"type":39,"tag":69,"props":2353,"children":2355},{"className":2354},[],[2356],{"type":45,"value":532},{"type":45,"value":2358}," to ",{"type":39,"tag":69,"props":2360,"children":2362},{"className":2361},[],[2363],{"type":45,"value":74},{"type":45,"value":2365}," and write a ",{"type":39,"tag":69,"props":2367,"children":2369},{"className":2368},[],[2370],{"type":45,"value":2371},"train()",{"type":45,"value":2373}," function that returns ",{"type":39,"tag":69,"props":2375,"children":2377},{"className":2376},[],[2378],{"type":45,"value":2379},"TrainingOutput(weights=Path(\"model.tar\"))",{"type":45,"value":2381},". The predictor then accepts the URL via ",{"type":39,"tag":69,"props":2383,"children":2385},{"className":2384},[],[2386],{"type":45,"value":2387},"setup(self, weights)",{"type":45,"value":2389}," or the ",{"type":39,"tag":69,"props":2391,"children":2393},{"className":2392},[],[2394],{"type":45,"value":2395},"COG_WEIGHTS",{"type":45,"value":2397}," env var. See ",{"type":39,"tag":58,"props":2399,"children":2402},{"href":2400,"rel":2401},"https:\u002F\u002Fcog.run\u002Ftraining",[62],[2403],{"type":45,"value":2400},{"type":45,"value":235},{"type":39,"tag":69,"props":2406,"children":2408},{"className":2407},[],[2409],{"type":45,"value":2410},"replicate\u002Fflux-fine-tuner",{"type":45,"value":2412}," for a full example.",{"type":39,"tag":40,"props":2414,"children":2416},{"id":2415},"guidelines",[2417],{"type":45,"value":2418},"Guidelines",{"type":39,"tag":48,"props":2420,"children":2421},{},[2422,2441,2454,2459,2477,2497,2508,2520],{"type":39,"tag":52,"props":2423,"children":2424},{},[2425,2427,2432,2434,2439],{"type":45,"value":2426},"Keep ",{"type":39,"tag":69,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":45,"value":1374},{"type":45,"value":2433}," for one-time loads; keep ",{"type":39,"tag":69,"props":2435,"children":2437},{"className":2436},[],[2438],{"type":45,"value":884},{"type":45,"value":2440}," fast and deterministic in shape.",{"type":39,"tag":52,"props":2442,"children":2443},{},[2444,2446,2452],{"type":45,"value":2445},"Pin Python and every dependency. Use ",{"type":39,"tag":69,"props":2447,"children":2449},{"className":2448},[],[2450],{"type":45,"value":2451},"numpy\u003C2",{"type":45,"value":2453}," if your torch is older.",{"type":39,"tag":52,"props":2455,"children":2456},{},[2457],{"type":45,"value":2458},"Always describe every input. Schemas without descriptions are unusable on the web UI.",{"type":39,"tag":52,"props":2460,"children":2461},{},[2462,2463,2468,2470,2475],{"type":45,"value":479},{"type":39,"tag":69,"props":2464,"children":2466},{"className":2465},[],[2467],{"type":45,"value":839},{"type":45,"value":2469}," for files and ",{"type":39,"tag":69,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":45,"value":851},{"type":45,"value":2476}," for tokens.",{"type":39,"tag":52,"props":2478,"children":2479},{},[2480,2482,2487,2489,2495],{"type":45,"value":2481},"Pin ",{"type":39,"tag":69,"props":2483,"children":2485},{"className":2484},[],[2486],{"type":45,"value":1297},{"type":45,"value":2488}," to a specific release (",{"type":39,"tag":69,"props":2490,"children":2492},{"className":2491},[],[2493],{"type":45,"value":2494},"v0.8.2",{"type":45,"value":2496},") for reproducibility.",{"type":39,"tag":52,"props":2498,"children":2499},{},[2500,2501,2506],{"type":45,"value":1545},{"type":39,"tag":69,"props":2502,"children":2504},{"className":2503},[],[2505],{"type":45,"value":1551},{"type":45,"value":2507}," whenever you call HuggingFace Hub.",{"type":39,"tag":52,"props":2509,"children":2510},{},[2511,2512,2518],{"type":45,"value":1545},{"type":39,"tag":69,"props":2513,"children":2515},{"className":2514},[],[2516],{"type":45,"value":2517},"TRANSFORMERS_OFFLINE=1",{"type":45,"value":2519}," after weights are loaded to prevent runtime HF lookups.",{"type":39,"tag":52,"props":2521,"children":2522},{},[2523,2525,2531],{"type":45,"value":2524},"Test with ",{"type":39,"tag":69,"props":2526,"children":2528},{"className":2527},[],[2529],{"type":45,"value":2530},"cog predict",{"type":45,"value":2532}," before pushing. If it doesn't work locally, it won't work in production.",{"type":39,"tag":40,"props":2534,"children":2536},{"id":2535},"production-references",[2537],{"type":45,"value":2538},"Production references",{"type":39,"tag":48,"props":2540,"children":2541},{},[2542,2552,2562,2573,2584,2595,2606,2617,2635],{"type":39,"tag":52,"props":2543,"children":2544},{},[2545,2550],{"type":39,"tag":58,"props":2546,"children":2548},{"href":101,"rel":2547},[62],[2549],{"type":45,"value":101},{"type":45,"value":2551}," — minimal patterns (resnet, hello-world, streaming, training)",{"type":39,"tag":52,"props":2553,"children":2554},{},[2555,2560],{"type":39,"tag":58,"props":2556,"children":2558},{"href":112,"rel":2557},[62],[2559],{"type":45,"value":112},{"type":45,"value":2561}," — scaffolder for new model repos",{"type":39,"tag":52,"props":2563,"children":2564},{},[2565,2571],{"type":39,"tag":58,"props":2566,"children":2569},{"href":2567,"rel":2568},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-flux",[62],[2570],{"type":45,"value":2567},{"type":45,"value":2572}," — multi-variant FLUX models, weights cache, fp8 + torch.compile",{"type":39,"tag":52,"props":2574,"children":2575},{},[2576,2582],{"type":39,"tag":58,"props":2577,"children":2580},{"href":2578,"rel":2579},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-flux-kontext",[62],[2581],{"type":45,"value":2578},{"type":45,"value":2583}," — meta-device loading, warmup compilation",{"type":39,"tag":52,"props":2585,"children":2586},{},[2587,2593],{"type":39,"tag":58,"props":2588,"children":2591},{"href":2589,"rel":2590},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-vllm",[62],[2592],{"type":45,"value":2589},{"type":45,"value":2594}," — async LLM server with continuous batching, training-as-packaging",{"type":39,"tag":52,"props":2596,"children":2597},{},[2598,2604],{"type":39,"tag":58,"props":2599,"children":2602},{"href":2600,"rel":2601},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fcog-comfyui",[62],[2603],{"type":45,"value":2600},{"type":45,"value":2605}," — ComfyUI workflows as a Cog model, custom-node helpers",{"type":39,"tag":52,"props":2607,"children":2608},{},[2609,2615],{"type":39,"tag":58,"props":2610,"children":2613},{"href":2611,"rel":2612},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fflux-fine-tuner",[62],[2614],{"type":45,"value":2611},{"type":45,"value":2616}," — multi-LoRA composition, shared pipeline components",{"type":39,"tag":52,"props":2618,"children":2619},{},[2620,2626,2628,2633],{"type":39,"tag":58,"props":2621,"children":2624},{"href":2622,"rel":2623},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fvibevoice",[62],[2625],{"type":45,"value":2622},{"type":45,"value":2627}," — TTS with dynamic ",{"type":39,"tag":69,"props":2629,"children":2631},{"className":2630},[],[2632],{"type":45,"value":871},{"type":45,"value":2634},", minimal cog.yaml",{"type":39,"tag":52,"props":2636,"children":2637},{},[2638,2644],{"type":39,"tag":58,"props":2639,"children":2642},{"href":2640,"rel":2641},"https:\u002F\u002Fgithub.com\u002Freplicate\u002Fpget",[62],[2643],{"type":45,"value":2640},{"type":45,"value":2645}," — parallel weights fetcher",{"type":39,"tag":2647,"props":2648,"children":2649},"style",{},[2650],{"type":45,"value":2651},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2653,"total":411},[2654,2660,2676,2686,2700,2712,2723],{"slug":4,"name":4,"fn":5,"description":6,"org":2655,"tags":2656,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2657,2658,2659],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":2661,"name":2661,"fn":2662,"description":2663,"org":2664,"tags":2665,"stars":22,"repoUrl":23,"updatedAt":2675},"compare-models","compare Replicate AI models","Compare Replicate models by cost, speed, quality, and capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2666,2669,2672],{"name":2667,"slug":2668,"type":15},"Analysis","analysis",{"name":2670,"slug":2671,"type":15},"Benchmarking","benchmarking",{"name":2673,"slug":2674,"type":15},"LLM","llm","2026-07-16T06:03:10.401096",{"slug":2677,"name":2677,"fn":2678,"description":2679,"org":2680,"tags":2681,"stars":22,"repoUrl":23,"updatedAt":2685},"find-models","find AI models on Replicate","Find AI models on Replicate using search and curated collections.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2682,2683,2684],{"name":20,"slug":21,"type":15},{"name":2673,"slug":2674,"type":15},{"name":13,"slug":14,"type":15},"2026-07-16T06:01:52.508562",{"slug":2687,"name":2687,"fn":2688,"description":2689,"org":2690,"tags":2691,"stars":22,"repoUrl":23,"updatedAt":2699},"prompt-images","generate prompts for image models","Prompting techniques for AI image generation and editing models on Replicate. Use when writing prompts for image models or building image generation features.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2692,2695,2698],{"name":2693,"slug":2694,"type":15},"Creative","creative",{"name":2696,"slug":2697,"type":15},"Image Generation","image-generation",{"name":2673,"slug":2674,"type":15},"2026-07-16T06:01:52.867443",{"slug":2701,"name":2701,"fn":2702,"description":2703,"org":2704,"tags":2705,"stars":22,"repoUrl":23,"updatedAt":2711},"prompt-videos","generate prompts for AI video models","Prompting techniques for AI video generation models on Replicate. Use when writing prompts for video models or building video generation features.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2706,2707,2708],{"name":2693,"slug":2694,"type":15},{"name":2673,"slug":2674,"type":15},{"name":2709,"slug":2710,"type":15},"Video","video","2026-07-16T06:03:10.065511",{"slug":168,"name":168,"fn":2713,"description":2714,"org":2715,"tags":2716,"stars":22,"repoUrl":23,"updatedAt":2722},"publish and deploy AI models to Replicate","Push and publish custom AI models to Replicate, and set up CI\u002FCD for releasing new model versions safely. Use when running cog push, deploying a model to Replicate, releasing a new version, validating a model with cog-safe-push before publishing, configuring a Replicate deployment, setting up GitHub Actions for model releases, or porting a community model to an official one. Trigger on phrases like \"push a model to Replicate\", \"publish a model\", \"deploy a model\", \"release a new version\", \"cog push\", \"cog-safe-push\", \"model CI\", \"r8.im\", or \"schema compatibility\", and when referencing github.com\u002Freplicate\u002Fcog-safe-push or github.com\u002Freplicate\u002Fmodel-ci-template. Covers cog push, the full cog-safe-push config (test cases, fuzz, deployment, official_model), GitHub Actions patterns, multi-model matrix pushes, and post-publish monitoring. Assumes you already have a working Cog project; see build-models if you need to package one first.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2717,2718,2721],{"name":20,"slug":21,"type":15},{"name":2719,"slug":2720,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-16T06:02:28.648245",{"slug":180,"name":180,"fn":2724,"description":2725,"org":2726,"tags":2727,"stars":22,"repoUrl":23,"updatedAt":2731},"run AI models on Replicate","Run AI models on Replicate via predictions, webhooks, and streaming.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2728,2729,2730],{"name":20,"slug":21,"type":15},{"name":2673,"slug":2674,"type":15},{"name":13,"slug":14,"type":15},"2026-07-16T06:03:09.738392",{"items":2733,"total":411},[2734,2740,2746,2752,2758,2764,2770],{"slug":4,"name":4,"fn":5,"description":6,"org":2735,"tags":2736,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2737,2738,2739],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":2661,"name":2661,"fn":2662,"description":2663,"org":2741,"tags":2742,"stars":22,"repoUrl":23,"updatedAt":2675},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2743,2744,2745],{"name":2667,"slug":2668,"type":15},{"name":2670,"slug":2671,"type":15},{"name":2673,"slug":2674,"type":15},{"slug":2677,"name":2677,"fn":2678,"description":2679,"org":2747,"tags":2748,"stars":22,"repoUrl":23,"updatedAt":2685},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2749,2750,2751],{"name":20,"slug":21,"type":15},{"name":2673,"slug":2674,"type":15},{"name":13,"slug":14,"type":15},{"slug":2687,"name":2687,"fn":2688,"description":2689,"org":2753,"tags":2754,"stars":22,"repoUrl":23,"updatedAt":2699},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2755,2756,2757],{"name":2693,"slug":2694,"type":15},{"name":2696,"slug":2697,"type":15},{"name":2673,"slug":2674,"type":15},{"slug":2701,"name":2701,"fn":2702,"description":2703,"org":2759,"tags":2760,"stars":22,"repoUrl":23,"updatedAt":2711},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2761,2762,2763],{"name":2693,"slug":2694,"type":15},{"name":2673,"slug":2674,"type":15},{"name":2709,"slug":2710,"type":15},{"slug":168,"name":168,"fn":2713,"description":2714,"org":2765,"tags":2766,"stars":22,"repoUrl":23,"updatedAt":2722},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2767,2768,2769],{"name":20,"slug":21,"type":15},{"name":2719,"slug":2720,"type":15},{"name":17,"slug":18,"type":15},{"slug":180,"name":180,"fn":2724,"description":2725,"org":2771,"tags":2772,"stars":22,"repoUrl":23,"updatedAt":2731},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2773,2774,2775],{"name":20,"slug":21,"type":15},{"name":2673,"slug":2674,"type":15},{"name":13,"slug":14,"type":15}]