[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-vllm-vllm-deploy-docker":3,"mdc-seuaaq-key":35,"related-org-vllm-vllm-deploy-docker":1999,"related-repo-vllm-vllm-deploy-docker":2092},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":33,"mdContent":34},"vllm-deploy-docker","deploy vLLM using Docker containers","Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"vllm","vLLM","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fvllm.png","vllm-project",[13,17,18,21],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":9,"slug":8,"type":16},{"name":19,"slug":20,"type":16},"Deployment","deployment",{"name":22,"slug":23,"type":16},"Docker","docker",88,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills","2026-07-18T05:47:01.814419",null,23,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":32},[],"Agent skills for vLLM","https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fvllm-skills\u002Fskills\u002Fvllm-deploy-docker","---\nname: vllm-deploy-docker\ndescription: Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.\n---\n\n# vLLM Docker Deployment\n\nA Claude skill describing how to deploy vLLM with Docker using the official pre-built images or building the image from source supporting NVIDIA GPUs with CUDA. Instructions include NVIDIA CUDA support, example `docker run` and a minimal `docker-compose` snippet, recommended flags, and troubleshooting notes. For AMD, Intel, or other accelerators, please refer to the [vLLM documentation](https:\u002F\u002Fdocs.vllm.ai\u002F) for alternative deployment methods.\n\n## What this skill does\n\n- Deploy vLLM with docker using pre-built images (recommended for most users) or build from source for custom configurations\n- Provide example commands for running the OpenAI-compatible server with GPU access and mounted Hugging Face cache\n- Point to build-from-source instructions when a custom image or optional dependencies are needed\n- Explain common flags: `--ipc=host`, shared cache mounts, and `HF_TOKEN` handling\n\n## Prerequisites\n\n- Docker Engine installed (Docker 20.10+ recommended)\n- NVIDIA GPU(s) with appropriate drivers and CUDA toolkit installed\n- Optional: `curl` for API tests\n- A Hugging Face token if pulling private models or to avoid rate-limits: `HF_TOKEN`\n\n## Quickstart using Pre-built Image (recommended)\n\nRun a vLLM OpenAI-compatible server with GPU access, mounting the HF cache and forwarding port 8000:\n\n```bash\ndocker run --rm --gpus all \\\n  -v ~\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  --env \"HF_TOKEN=$HF_TOKEN\" \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  vllm\u002Fvllm-openai:latest \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n```\n\n- `--gpus all` exposes all GPUs to the container. Adjust if you need specific GPUs.\n- `--ipc=host` or an appropriately large `--shm-size` is recommended so PyTorch and vLLM can share host shared memory.\n- Mounting `~\u002F.cache\u002Fhuggingface` avoids re-downloading models inside the container.\n\n> **Note:** vLLM and this skill recommend using the latest Docker image (`vllm\u002Fvllm-openai:latest`). For legacy version images, you may refer to the [Docker Hub image tags](https:\u002F\u002Fhub.docker.com\u002Fr\u002Fvllm\u002Fvllm-openai\u002Ftags).\n\n## Build Docker image from source\n\nYou can build and run vLLM from source by using the provided [docker\u002FDockerfile](https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fblob\u002Fmain\u002Fdocker\u002FDockerfile). \nFirst, check the hardware of the host machine and ensure you have the necessary dependencies installed (e.g., NVIDIA drivers, CUDA toolkit, Docker with BuildKit support). For ARM64\u002Faarch64 builds, refer to the \"Building for ARM64\u002Faarch64\" section.\n\n### Basic build command\n\n```bash\nDOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile\n```\n\nThe `--target vllm-openai` specifies that you are building the OpenAI-compatible server image. The `DOCKER_BUILDKIT=1` environment variable enables BuildKit, which provides better caching and faster builds.\n\n### Build arguments and options\n\n- **`--build-arg max_jobs=\u003CN>`** — sets the number of parallel compilation jobs for building CUDA kernels. Useful for speeding up builds on multi-core systems.\n- **`--build-arg nvcc_threads=\u003CN>`** — controls CUDA compiler threads. Recommended to use a smaller value than `max_jobs` to avoid excessive memory usage.\n- **`--build-arg torch_cuda_arch_list=\"\"`** — if set to empty string, vLLM will detect and build only for the current GPU's compute capability. By default, vLLM builds for all GPU types for wider distribution.\n\n### Using precompiled wheels to speed up builds\n\nIf you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time:\n\n- **Enable precompiled wheels:** Add `--build-arg VLLM_USE_PRECOMPILED=\"1\"` to your build command.\n- **How it works:** By default, vLLM automatically finds the correct precompiled wheels from the [Nightly Builds](https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcontributing\u002Fci\u002Fnightly_builds\u002F) by using the merge-base commit with the upstream `main` branch.\n- **Specify a commit:** To use wheels from a specific commit, add `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=\u003Ccommit_hash>`.\n\n**Example with precompiled wheels and options for fast compilation:**\n```bash\nDOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile \\\n  --build-arg max_jobs=8 \\\n  --build-arg nvcc_threads=2 \\\n  --build-arg VLLM_USE_PRECOMPILED=\"1\"\n```\n\n### Building with optional dependencies (optional)\n\nvLLM does not include optional dependencies (e.g., audio processing) in the pre-built image to avoid licensing issues. If you need optional dependencies, create a custom Dockerfile that extends the base image:\n\n**Example: adding audio optional dependencies**\n```dockerfile\n# NOTE: MAKE SURE the version of vLLM matches the base image!\nFROM vllm\u002Fvllm-openai:0.11.0\n\n# Install audio optional dependencies\nRUN uv pip install --system vllm[audio]==0.11.0\n```\n\n**Example: using development version of transformers:**\n```dockerfile\nFROM vllm\u002Fvllm-openai:latest\n\n# Install development version of Transformers from source\nRUN uv pip install --system git+https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers.git\n```\n\nBuild this custom Dockerfile with:\n```bash\ndocker build -t my-vllm-custom:latest -f Dockerfile .\n```\n\nThen use it like any other vLLM image:\n```bash\ndocker run --rm --gpus all \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  my-vllm-custom:latest \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n```\n\n### Building for ARM64\u002Faarch64\n\nA Docker container can be built for ARM64 systems (e.g., NVIDIA Grace-Hopper and Grace-Blackwell). Use the flag `--platform \"linux\u002Farm64\"`:\n\n```bash\nDOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile \\\n  --platform \"linux\u002Farm64\"\n```\n\n**Note:** Multiple modules must be compiled, so this process can take longer. Use build arguments like `--build-arg max_jobs=8 --build-arg nvcc_threads=2` to speed up the process (ensure `max_jobs` is substantially larger than `nvcc_threads`). Monitor memory usage, as parallel jobs can require significant RAM.\n\n**For cross-compilation** (building ARM64 on an x86_64 host), register QEMU user-static handlers first:\n```bash\ndocker run --rm --privileged multiarch\u002Fqemu-user-static --reset -p yes\n```\n\nThen use the `--platform \"linux\u002Farm64\"` flag in your build command.\n\n### Running your custom-built image\n\nAfter building, run your image just like the pre-built image:\n\n```bash\ndocker run --rm --runtime nvidia --gpus all \\\n  -v ~\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  --env \"HF_TOKEN=$HF_TOKEN\" \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  vllm\u002Fvllm-openai \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n```\n\nReplace `vllm\u002Fvllm-openai` with the tag you specified during the build (e.g., `my-vllm-custom:latest`).\n> **Note:** `--runtime nvidia` is deprecated for most environments. Prefer `--gpus ...` with NVIDIA Container Toolkit. Use `--runtime nvidia` only for legacy Docker configurations.\n\n## Common server flags\n\n- `--model \u003CMODEL_ID>` — model to load (HF ID or local path)\n- `--port \u003CPORT>` — server port (default 8000 for OpenAI-compatible server)\n- `--log-level` — adjust verbosity\n- You may pass additional `engine_args` after the image tag; see vLLM docs for tuning options.\n\n## Testing the API\n\nAfter the container starts, make a quick test request against the OpenAI-compatible endpoint:\n\n```bash\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"model\":\"Qwen\u002FQwen2.5-1.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Who are you?\"}],\"max_tokens\":128}'\n```\n\n## Security and operational notes\n\n- Keep `HF_TOKEN` secret; prefer passing it via environment variables or a secret manager.\n- For production, run behind a reverse proxy (Nginx) with TLS and authentication.\n- Mount only necessary host paths into the container.\n\n## Troubleshooting\n\n- Container can't access GPUs: ensure `nvidia-container-toolkit` is installed and restart Docker.\n- Model download failures: check `HF_TOKEN` and network; mount cache directory to persist downloads.\n- Memory \u002F OOM errors: try a smaller model or add more GPU memory; check `--shm-size`.\n- If the container fails with NCCL library path issues (rare): set `VLLM_NCCL_SO_PATH` per upstream guidance.\n- Permission issues of current user with Docker: ensure your user is in the `docker` group or suggest user to add current user to `docker` group manually following:\n```bash\n# 1. Create docker group if it doesn't exist (may already exist on some systems)\nsudo groupadd docker\n\n# 2. Add current user to the docker group (replace $USER with your username if needed)\nsudo usermod -aG docker $USER\n\n# 3. Apply the new group membership (you may need to log out and log back in for this to take effect)\nnewgrp docker\n\n# 4. Verify that the user is in the docker group (output should include docker)\ngroups $USER\n```\n- Docker pull failed due to a network timeout connecting to Docker Hub: check the network connection and try again. If the issue persists, suggest and show the user how to use a mirror for Docker Hub.\n- Error during downloading models from Hugging Face: check if the `HF_TOKEN` is passed to the container and is valid. Check if `HTTP_PROXY` and `HTTPS_PROXY` are passed to the container if the host is behind a proxy. Also, verify that the model ID is correct and that the model is public or accessible with the provided token.\n\n## References\n\n- vLLM repository (docker\u002FDockerfile): https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Ftree\u002Fmain\u002Fdocker\n- NVIDIA Container Toolkit: https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fcontainer-toolkit\u002Flatest\u002Finstall-guide.html\n- Up-to-date deployment instructions and troubleshooting: https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fdeployment\u002Fdocker\u002F",{"data":36,"body":37},{"name":4,"description":6},{"type":38,"children":39},"root",[40,49,83,90,131,137,173,179,184,343,388,420,426,440,447,538,559,565,618,624,629,694,702,855,861,866,874,924,932,970,975,1018,1023,1106,1112,1125,1229,1261,1271,1318,1330,1336,1341,1476,1497,1532,1538,1587,1593,1598,1674,1680,1705,1711,1782,1915,1951,1957,1993],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"vllm-docker-deployment",[46],{"type":47,"value":48},"text","vLLM Docker Deployment",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53,55,62,64,70,72,81],{"type":47,"value":54},"A Claude skill describing how to deploy vLLM with Docker using the official pre-built images or building the image from source supporting NVIDIA GPUs with CUDA. Instructions include NVIDIA CUDA support, example ",{"type":41,"tag":56,"props":57,"children":59},"code",{"className":58},[],[60],{"type":47,"value":61},"docker run",{"type":47,"value":63}," and a minimal ",{"type":41,"tag":56,"props":65,"children":67},{"className":66},[],[68],{"type":47,"value":69},"docker-compose",{"type":47,"value":71}," snippet, recommended flags, and troubleshooting notes. For AMD, Intel, or other accelerators, please refer to the ",{"type":41,"tag":73,"props":74,"children":78},"a",{"href":75,"rel":76},"https:\u002F\u002Fdocs.vllm.ai\u002F",[77],"nofollow",[79],{"type":47,"value":80},"vLLM documentation",{"type":47,"value":82}," for alternative deployment methods.",{"type":41,"tag":84,"props":85,"children":87},"h2",{"id":86},"what-this-skill-does",[88],{"type":47,"value":89},"What this skill does",{"type":41,"tag":91,"props":92,"children":93},"ul",{},[94,100,105,110],{"type":41,"tag":95,"props":96,"children":97},"li",{},[98],{"type":47,"value":99},"Deploy vLLM with docker using pre-built images (recommended for most users) or build from source for custom configurations",{"type":41,"tag":95,"props":101,"children":102},{},[103],{"type":47,"value":104},"Provide example commands for running the OpenAI-compatible server with GPU access and mounted Hugging Face cache",{"type":41,"tag":95,"props":106,"children":107},{},[108],{"type":47,"value":109},"Point to build-from-source instructions when a custom image or optional dependencies are needed",{"type":41,"tag":95,"props":111,"children":112},{},[113,115,121,123,129],{"type":47,"value":114},"Explain common flags: ",{"type":41,"tag":56,"props":116,"children":118},{"className":117},[],[119],{"type":47,"value":120},"--ipc=host",{"type":47,"value":122},", shared cache mounts, and ",{"type":41,"tag":56,"props":124,"children":126},{"className":125},[],[127],{"type":47,"value":128},"HF_TOKEN",{"type":47,"value":130}," handling",{"type":41,"tag":84,"props":132,"children":134},{"id":133},"prerequisites",[135],{"type":47,"value":136},"Prerequisites",{"type":41,"tag":91,"props":138,"children":139},{},[140,145,150,163],{"type":41,"tag":95,"props":141,"children":142},{},[143],{"type":47,"value":144},"Docker Engine installed (Docker 20.10+ recommended)",{"type":41,"tag":95,"props":146,"children":147},{},[148],{"type":47,"value":149},"NVIDIA GPU(s) with appropriate drivers and CUDA toolkit installed",{"type":41,"tag":95,"props":151,"children":152},{},[153,155,161],{"type":47,"value":154},"Optional: ",{"type":41,"tag":56,"props":156,"children":158},{"className":157},[],[159],{"type":47,"value":160},"curl",{"type":47,"value":162}," for API tests",{"type":41,"tag":95,"props":164,"children":165},{},[166,168],{"type":47,"value":167},"A Hugging Face token if pulling private models or to avoid rate-limits: ",{"type":41,"tag":56,"props":169,"children":171},{"className":170},[],[172],{"type":47,"value":128},{"type":41,"tag":84,"props":174,"children":176},{"id":175},"quickstart-using-pre-built-image-recommended",[177],{"type":47,"value":178},"Quickstart using Pre-built Image (recommended)",{"type":41,"tag":50,"props":180,"children":181},{},[182],{"type":47,"value":183},"Run a vLLM OpenAI-compatible server with GPU access, mounting the HF cache and forwarding port 8000:",{"type":41,"tag":185,"props":186,"children":191},"pre",{"className":187,"code":188,"language":189,"meta":190,"style":190},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","docker run --rm --gpus all \\\n  -v ~\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  --env \"HF_TOKEN=$HF_TOKEN\" \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  vllm\u002Fvllm-openai:latest \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n","bash","",[192],{"type":41,"tag":56,"props":193,"children":194},{"__ignoreMap":190},[195,233,251,285,303,316,329],{"type":41,"tag":196,"props":197,"children":200},"span",{"class":198,"line":199},"line",1,[201,206,212,217,222,227],{"type":41,"tag":196,"props":202,"children":204},{"style":203},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[205],{"type":47,"value":23},{"type":41,"tag":196,"props":207,"children":209},{"style":208},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[210],{"type":47,"value":211}," run",{"type":41,"tag":196,"props":213,"children":214},{"style":208},[215],{"type":47,"value":216}," --rm",{"type":41,"tag":196,"props":218,"children":219},{"style":208},[220],{"type":47,"value":221}," --gpus",{"type":41,"tag":196,"props":223,"children":224},{"style":208},[225],{"type":47,"value":226}," all",{"type":41,"tag":196,"props":228,"children":230},{"style":229},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[231],{"type":47,"value":232}," \\\n",{"type":41,"tag":196,"props":234,"children":236},{"class":198,"line":235},2,[237,242,247],{"type":41,"tag":196,"props":238,"children":239},{"style":208},[240],{"type":47,"value":241},"  -v",{"type":41,"tag":196,"props":243,"children":244},{"style":208},[245],{"type":47,"value":246}," ~\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface",{"type":41,"tag":196,"props":248,"children":249},{"style":229},[250],{"type":47,"value":232},{"type":41,"tag":196,"props":252,"children":254},{"class":198,"line":253},3,[255,260,266,271,276,281],{"type":41,"tag":196,"props":256,"children":257},{"style":208},[258],{"type":47,"value":259},"  --env",{"type":41,"tag":196,"props":261,"children":263},{"style":262},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[264],{"type":47,"value":265}," \"",{"type":41,"tag":196,"props":267,"children":268},{"style":208},[269],{"type":47,"value":270},"HF_TOKEN=",{"type":41,"tag":196,"props":272,"children":273},{"style":229},[274],{"type":47,"value":275},"$HF_TOKEN",{"type":41,"tag":196,"props":277,"children":278},{"style":262},[279],{"type":47,"value":280},"\"",{"type":41,"tag":196,"props":282,"children":283},{"style":229},[284],{"type":47,"value":232},{"type":41,"tag":196,"props":286,"children":288},{"class":198,"line":287},4,[289,294,299],{"type":41,"tag":196,"props":290,"children":291},{"style":208},[292],{"type":47,"value":293},"  -p",{"type":41,"tag":196,"props":295,"children":296},{"style":208},[297],{"type":47,"value":298}," 8000:8000",{"type":41,"tag":196,"props":300,"children":301},{"style":229},[302],{"type":47,"value":232},{"type":41,"tag":196,"props":304,"children":306},{"class":198,"line":305},5,[307,312],{"type":41,"tag":196,"props":308,"children":309},{"style":208},[310],{"type":47,"value":311},"  --ipc=host",{"type":41,"tag":196,"props":313,"children":314},{"style":229},[315],{"type":47,"value":232},{"type":41,"tag":196,"props":317,"children":319},{"class":198,"line":318},6,[320,325],{"type":41,"tag":196,"props":321,"children":322},{"style":208},[323],{"type":47,"value":324},"  vllm\u002Fvllm-openai:latest",{"type":41,"tag":196,"props":326,"children":327},{"style":229},[328],{"type":47,"value":232},{"type":41,"tag":196,"props":330,"children":332},{"class":198,"line":331},7,[333,338],{"type":41,"tag":196,"props":334,"children":335},{"style":208},[336],{"type":47,"value":337},"  --model",{"type":41,"tag":196,"props":339,"children":340},{"style":208},[341],{"type":47,"value":342}," Qwen\u002FQwen2.5-1.5B-Instruct\n",{"type":41,"tag":91,"props":344,"children":345},{},[346,357,375],{"type":41,"tag":95,"props":347,"children":348},{},[349,355],{"type":41,"tag":56,"props":350,"children":352},{"className":351},[],[353],{"type":47,"value":354},"--gpus all",{"type":47,"value":356}," exposes all GPUs to the container. Adjust if you need specific GPUs.",{"type":41,"tag":95,"props":358,"children":359},{},[360,365,367,373],{"type":41,"tag":56,"props":361,"children":363},{"className":362},[],[364],{"type":47,"value":120},{"type":47,"value":366}," or an appropriately large ",{"type":41,"tag":56,"props":368,"children":370},{"className":369},[],[371],{"type":47,"value":372},"--shm-size",{"type":47,"value":374}," is recommended so PyTorch and vLLM can share host shared memory.",{"type":41,"tag":95,"props":376,"children":377},{},[378,380,386],{"type":47,"value":379},"Mounting ",{"type":41,"tag":56,"props":381,"children":383},{"className":382},[],[384],{"type":47,"value":385},"~\u002F.cache\u002Fhuggingface",{"type":47,"value":387}," avoids re-downloading models inside the container.",{"type":41,"tag":389,"props":390,"children":391},"blockquote",{},[392],{"type":41,"tag":50,"props":393,"children":394},{},[395,401,403,409,411,418],{"type":41,"tag":396,"props":397,"children":398},"strong",{},[399],{"type":47,"value":400},"Note:",{"type":47,"value":402}," vLLM and this skill recommend using the latest Docker image (",{"type":41,"tag":56,"props":404,"children":406},{"className":405},[],[407],{"type":47,"value":408},"vllm\u002Fvllm-openai:latest",{"type":47,"value":410},"). For legacy version images, you may refer to the ",{"type":41,"tag":73,"props":412,"children":415},{"href":413,"rel":414},"https:\u002F\u002Fhub.docker.com\u002Fr\u002Fvllm\u002Fvllm-openai\u002Ftags",[77],[416],{"type":47,"value":417},"Docker Hub image tags",{"type":47,"value":419},".",{"type":41,"tag":84,"props":421,"children":423},{"id":422},"build-docker-image-from-source",[424],{"type":47,"value":425},"Build Docker image from source",{"type":41,"tag":50,"props":427,"children":428},{},[429,431,438],{"type":47,"value":430},"You can build and run vLLM from source by using the provided ",{"type":41,"tag":73,"props":432,"children":435},{"href":433,"rel":434},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fblob\u002Fmain\u002Fdocker\u002FDockerfile",[77],[436],{"type":47,"value":437},"docker\u002FDockerfile",{"type":47,"value":439},".\nFirst, check the hardware of the host machine and ensure you have the necessary dependencies installed (e.g., NVIDIA drivers, CUDA toolkit, Docker with BuildKit support). For ARM64\u002Faarch64 builds, refer to the \"Building for ARM64\u002Faarch64\" section.",{"type":41,"tag":441,"props":442,"children":444},"h3",{"id":443},"basic-build-command",[445],{"type":47,"value":446},"Basic build command",{"type":41,"tag":185,"props":448,"children":450},{"className":187,"code":449,"language":189,"meta":190,"style":190},"DOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile\n",[451],{"type":41,"tag":56,"props":452,"children":453},{"__ignoreMap":190},[454,491,508,525],{"type":41,"tag":196,"props":455,"children":456},{"class":198,"line":199},[457,462,467,472,477,482,487],{"type":41,"tag":196,"props":458,"children":459},{"style":229},[460],{"type":47,"value":461},"DOCKER_BUILDKIT",{"type":41,"tag":196,"props":463,"children":464},{"style":262},[465],{"type":47,"value":466},"=",{"type":41,"tag":196,"props":468,"children":469},{"style":208},[470],{"type":47,"value":471},"1",{"type":41,"tag":196,"props":473,"children":474},{"style":203},[475],{"type":47,"value":476}," docker",{"type":41,"tag":196,"props":478,"children":479},{"style":208},[480],{"type":47,"value":481}," build",{"type":41,"tag":196,"props":483,"children":484},{"style":208},[485],{"type":47,"value":486}," .",{"type":41,"tag":196,"props":488,"children":489},{"style":229},[490],{"type":47,"value":232},{"type":41,"tag":196,"props":492,"children":493},{"class":198,"line":235},[494,499,504],{"type":41,"tag":196,"props":495,"children":496},{"style":208},[497],{"type":47,"value":498},"  --target",{"type":41,"tag":196,"props":500,"children":501},{"style":208},[502],{"type":47,"value":503}," vllm-openai",{"type":41,"tag":196,"props":505,"children":506},{"style":229},[507],{"type":47,"value":232},{"type":41,"tag":196,"props":509,"children":510},{"class":198,"line":253},[511,516,521],{"type":41,"tag":196,"props":512,"children":513},{"style":208},[514],{"type":47,"value":515},"  --tag",{"type":41,"tag":196,"props":517,"children":518},{"style":208},[519],{"type":47,"value":520}," vllm\u002Fvllm-openai",{"type":41,"tag":196,"props":522,"children":523},{"style":229},[524],{"type":47,"value":232},{"type":41,"tag":196,"props":526,"children":527},{"class":198,"line":287},[528,533],{"type":41,"tag":196,"props":529,"children":530},{"style":208},[531],{"type":47,"value":532},"  --file",{"type":41,"tag":196,"props":534,"children":535},{"style":208},[536],{"type":47,"value":537}," docker\u002FDockerfile\n",{"type":41,"tag":50,"props":539,"children":540},{},[541,543,549,551,557],{"type":47,"value":542},"The ",{"type":41,"tag":56,"props":544,"children":546},{"className":545},[],[547],{"type":47,"value":548},"--target vllm-openai",{"type":47,"value":550}," specifies that you are building the OpenAI-compatible server image. The ",{"type":41,"tag":56,"props":552,"children":554},{"className":553},[],[555],{"type":47,"value":556},"DOCKER_BUILDKIT=1",{"type":47,"value":558}," environment variable enables BuildKit, which provides better caching and faster builds.",{"type":41,"tag":441,"props":560,"children":562},{"id":561},"build-arguments-and-options",[563],{"type":47,"value":564},"Build arguments and options",{"type":41,"tag":91,"props":566,"children":567},{},[568,582,604],{"type":41,"tag":95,"props":569,"children":570},{},[571,580],{"type":41,"tag":396,"props":572,"children":573},{},[574],{"type":41,"tag":56,"props":575,"children":577},{"className":576},[],[578],{"type":47,"value":579},"--build-arg max_jobs=\u003CN>",{"type":47,"value":581}," — sets the number of parallel compilation jobs for building CUDA kernels. Useful for speeding up builds on multi-core systems.",{"type":41,"tag":95,"props":583,"children":584},{},[585,594,596,602],{"type":41,"tag":396,"props":586,"children":587},{},[588],{"type":41,"tag":56,"props":589,"children":591},{"className":590},[],[592],{"type":47,"value":593},"--build-arg nvcc_threads=\u003CN>",{"type":47,"value":595}," — controls CUDA compiler threads. Recommended to use a smaller value than ",{"type":41,"tag":56,"props":597,"children":599},{"className":598},[],[600],{"type":47,"value":601},"max_jobs",{"type":47,"value":603}," to avoid excessive memory usage.",{"type":41,"tag":95,"props":605,"children":606},{},[607,616],{"type":41,"tag":396,"props":608,"children":609},{},[610],{"type":41,"tag":56,"props":611,"children":613},{"className":612},[],[614],{"type":47,"value":615},"--build-arg torch_cuda_arch_list=\"\"",{"type":47,"value":617}," — if set to empty string, vLLM will detect and build only for the current GPU's compute capability. By default, vLLM builds for all GPU types for wider distribution.",{"type":41,"tag":441,"props":619,"children":621},{"id":620},"using-precompiled-wheels-to-speed-up-builds",[622],{"type":47,"value":623},"Using precompiled wheels to speed up builds",{"type":41,"tag":50,"props":625,"children":626},{},[627],{"type":47,"value":628},"If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time:",{"type":41,"tag":91,"props":630,"children":631},{},[632,650,677],{"type":41,"tag":95,"props":633,"children":634},{},[635,640,642,648],{"type":41,"tag":396,"props":636,"children":637},{},[638],{"type":47,"value":639},"Enable precompiled wheels:",{"type":47,"value":641}," Add ",{"type":41,"tag":56,"props":643,"children":645},{"className":644},[],[646],{"type":47,"value":647},"--build-arg VLLM_USE_PRECOMPILED=\"1\"",{"type":47,"value":649}," to your build command.",{"type":41,"tag":95,"props":651,"children":652},{},[653,658,660,667,669,675],{"type":41,"tag":396,"props":654,"children":655},{},[656],{"type":47,"value":657},"How it works:",{"type":47,"value":659}," By default, vLLM automatically finds the correct precompiled wheels from the ",{"type":41,"tag":73,"props":661,"children":664},{"href":662,"rel":663},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcontributing\u002Fci\u002Fnightly_builds\u002F",[77],[665],{"type":47,"value":666},"Nightly Builds",{"type":47,"value":668}," by using the merge-base commit with the upstream ",{"type":41,"tag":56,"props":670,"children":672},{"className":671},[],[673],{"type":47,"value":674},"main",{"type":47,"value":676}," branch.",{"type":41,"tag":95,"props":678,"children":679},{},[680,685,687,693],{"type":41,"tag":396,"props":681,"children":682},{},[683],{"type":47,"value":684},"Specify a commit:",{"type":47,"value":686}," To use wheels from a specific commit, add ",{"type":41,"tag":56,"props":688,"children":690},{"className":689},[],[691],{"type":47,"value":692},"--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=\u003Ccommit_hash>",{"type":47,"value":419},{"type":41,"tag":50,"props":695,"children":696},{},[697],{"type":41,"tag":396,"props":698,"children":699},{},[700],{"type":47,"value":701},"Example with precompiled wheels and options for fast compilation:",{"type":41,"tag":185,"props":703,"children":705},{"className":187,"code":704,"language":189,"meta":190,"style":190},"DOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile \\\n  --build-arg max_jobs=8 \\\n  --build-arg nvcc_threads=2 \\\n  --build-arg VLLM_USE_PRECOMPILED=\"1\"\n",[706],{"type":41,"tag":56,"props":707,"children":708},{"__ignoreMap":190},[709,740,755,770,786,809,830],{"type":41,"tag":196,"props":710,"children":711},{"class":198,"line":199},[712,716,720,724,728,732,736],{"type":41,"tag":196,"props":713,"children":714},{"style":229},[715],{"type":47,"value":461},{"type":41,"tag":196,"props":717,"children":718},{"style":262},[719],{"type":47,"value":466},{"type":41,"tag":196,"props":721,"children":722},{"style":208},[723],{"type":47,"value":471},{"type":41,"tag":196,"props":725,"children":726},{"style":203},[727],{"type":47,"value":476},{"type":41,"tag":196,"props":729,"children":730},{"style":208},[731],{"type":47,"value":481},{"type":41,"tag":196,"props":733,"children":734},{"style":208},[735],{"type":47,"value":486},{"type":41,"tag":196,"props":737,"children":738},{"style":229},[739],{"type":47,"value":232},{"type":41,"tag":196,"props":741,"children":742},{"class":198,"line":235},[743,747,751],{"type":41,"tag":196,"props":744,"children":745},{"style":208},[746],{"type":47,"value":498},{"type":41,"tag":196,"props":748,"children":749},{"style":208},[750],{"type":47,"value":503},{"type":41,"tag":196,"props":752,"children":753},{"style":229},[754],{"type":47,"value":232},{"type":41,"tag":196,"props":756,"children":757},{"class":198,"line":253},[758,762,766],{"type":41,"tag":196,"props":759,"children":760},{"style":208},[761],{"type":47,"value":515},{"type":41,"tag":196,"props":763,"children":764},{"style":208},[765],{"type":47,"value":520},{"type":41,"tag":196,"props":767,"children":768},{"style":229},[769],{"type":47,"value":232},{"type":41,"tag":196,"props":771,"children":772},{"class":198,"line":287},[773,777,782],{"type":41,"tag":196,"props":774,"children":775},{"style":208},[776],{"type":47,"value":532},{"type":41,"tag":196,"props":778,"children":779},{"style":208},[780],{"type":47,"value":781}," docker\u002FDockerfile",{"type":41,"tag":196,"props":783,"children":784},{"style":229},[785],{"type":47,"value":232},{"type":41,"tag":196,"props":787,"children":788},{"class":198,"line":305},[789,794,799,805],{"type":41,"tag":196,"props":790,"children":791},{"style":208},[792],{"type":47,"value":793},"  --build-arg",{"type":41,"tag":196,"props":795,"children":796},{"style":208},[797],{"type":47,"value":798}," max_jobs=",{"type":41,"tag":196,"props":800,"children":802},{"style":801},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[803],{"type":47,"value":804},"8",{"type":41,"tag":196,"props":806,"children":807},{"style":229},[808],{"type":47,"value":232},{"type":41,"tag":196,"props":810,"children":811},{"class":198,"line":318},[812,816,821,826],{"type":41,"tag":196,"props":813,"children":814},{"style":208},[815],{"type":47,"value":793},{"type":41,"tag":196,"props":817,"children":818},{"style":208},[819],{"type":47,"value":820}," nvcc_threads=",{"type":41,"tag":196,"props":822,"children":823},{"style":801},[824],{"type":47,"value":825},"2",{"type":41,"tag":196,"props":827,"children":828},{"style":229},[829],{"type":47,"value":232},{"type":41,"tag":196,"props":831,"children":832},{"class":198,"line":331},[833,837,842,846,850],{"type":41,"tag":196,"props":834,"children":835},{"style":208},[836],{"type":47,"value":793},{"type":41,"tag":196,"props":838,"children":839},{"style":208},[840],{"type":47,"value":841}," VLLM_USE_PRECOMPILED=",{"type":41,"tag":196,"props":843,"children":844},{"style":262},[845],{"type":47,"value":280},{"type":41,"tag":196,"props":847,"children":848},{"style":208},[849],{"type":47,"value":471},{"type":41,"tag":196,"props":851,"children":852},{"style":262},[853],{"type":47,"value":854},"\"\n",{"type":41,"tag":441,"props":856,"children":858},{"id":857},"building-with-optional-dependencies-optional",[859],{"type":47,"value":860},"Building with optional dependencies (optional)",{"type":41,"tag":50,"props":862,"children":863},{},[864],{"type":47,"value":865},"vLLM does not include optional dependencies (e.g., audio processing) in the pre-built image to avoid licensing issues. If you need optional dependencies, create a custom Dockerfile that extends the base image:",{"type":41,"tag":50,"props":867,"children":868},{},[869],{"type":41,"tag":396,"props":870,"children":871},{},[872],{"type":47,"value":873},"Example: adding audio optional dependencies",{"type":41,"tag":185,"props":875,"children":879},{"className":876,"code":877,"language":878,"meta":190,"style":190},"language-dockerfile shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# NOTE: MAKE SURE the version of vLLM matches the base image!\nFROM vllm\u002Fvllm-openai:0.11.0\n\n# Install audio optional dependencies\nRUN uv pip install --system vllm[audio]==0.11.0\n","dockerfile",[880],{"type":41,"tag":56,"props":881,"children":882},{"__ignoreMap":190},[883,891,899,908,916],{"type":41,"tag":196,"props":884,"children":885},{"class":198,"line":199},[886],{"type":41,"tag":196,"props":887,"children":888},{},[889],{"type":47,"value":890},"# NOTE: MAKE SURE the version of vLLM matches the base image!\n",{"type":41,"tag":196,"props":892,"children":893},{"class":198,"line":235},[894],{"type":41,"tag":196,"props":895,"children":896},{},[897],{"type":47,"value":898},"FROM vllm\u002Fvllm-openai:0.11.0\n",{"type":41,"tag":196,"props":900,"children":901},{"class":198,"line":253},[902],{"type":41,"tag":196,"props":903,"children":905},{"emptyLinePlaceholder":904},true,[906],{"type":47,"value":907},"\n",{"type":41,"tag":196,"props":909,"children":910},{"class":198,"line":287},[911],{"type":41,"tag":196,"props":912,"children":913},{},[914],{"type":47,"value":915},"# Install audio optional dependencies\n",{"type":41,"tag":196,"props":917,"children":918},{"class":198,"line":305},[919],{"type":41,"tag":196,"props":920,"children":921},{},[922],{"type":47,"value":923},"RUN uv pip install --system vllm[audio]==0.11.0\n",{"type":41,"tag":50,"props":925,"children":926},{},[927],{"type":41,"tag":396,"props":928,"children":929},{},[930],{"type":47,"value":931},"Example: using development version of transformers:",{"type":41,"tag":185,"props":933,"children":935},{"className":876,"code":934,"language":878,"meta":190,"style":190},"FROM vllm\u002Fvllm-openai:latest\n\n# Install development version of Transformers from source\nRUN uv pip install --system git+https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers.git\n",[936],{"type":41,"tag":56,"props":937,"children":938},{"__ignoreMap":190},[939,947,954,962],{"type":41,"tag":196,"props":940,"children":941},{"class":198,"line":199},[942],{"type":41,"tag":196,"props":943,"children":944},{},[945],{"type":47,"value":946},"FROM vllm\u002Fvllm-openai:latest\n",{"type":41,"tag":196,"props":948,"children":949},{"class":198,"line":235},[950],{"type":41,"tag":196,"props":951,"children":952},{"emptyLinePlaceholder":904},[953],{"type":47,"value":907},{"type":41,"tag":196,"props":955,"children":956},{"class":198,"line":253},[957],{"type":41,"tag":196,"props":958,"children":959},{},[960],{"type":47,"value":961},"# Install development version of Transformers from source\n",{"type":41,"tag":196,"props":963,"children":964},{"class":198,"line":287},[965],{"type":41,"tag":196,"props":966,"children":967},{},[968],{"type":47,"value":969},"RUN uv pip install --system git+https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers.git\n",{"type":41,"tag":50,"props":971,"children":972},{},[973],{"type":47,"value":974},"Build this custom Dockerfile with:",{"type":41,"tag":185,"props":976,"children":978},{"className":187,"code":977,"language":189,"meta":190,"style":190},"docker build -t my-vllm-custom:latest -f Dockerfile .\n",[979],{"type":41,"tag":56,"props":980,"children":981},{"__ignoreMap":190},[982],{"type":41,"tag":196,"props":983,"children":984},{"class":198,"line":199},[985,989,993,998,1003,1008,1013],{"type":41,"tag":196,"props":986,"children":987},{"style":203},[988],{"type":47,"value":23},{"type":41,"tag":196,"props":990,"children":991},{"style":208},[992],{"type":47,"value":481},{"type":41,"tag":196,"props":994,"children":995},{"style":208},[996],{"type":47,"value":997}," -t",{"type":41,"tag":196,"props":999,"children":1000},{"style":208},[1001],{"type":47,"value":1002}," my-vllm-custom:latest",{"type":41,"tag":196,"props":1004,"children":1005},{"style":208},[1006],{"type":47,"value":1007}," -f",{"type":41,"tag":196,"props":1009,"children":1010},{"style":208},[1011],{"type":47,"value":1012}," Dockerfile",{"type":41,"tag":196,"props":1014,"children":1015},{"style":208},[1016],{"type":47,"value":1017}," .\n",{"type":41,"tag":50,"props":1019,"children":1020},{},[1021],{"type":47,"value":1022},"Then use it like any other vLLM image:",{"type":41,"tag":185,"props":1024,"children":1026},{"className":187,"code":1025,"language":189,"meta":190,"style":190},"docker run --rm --gpus all \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  my-vllm-custom:latest \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n",[1027],{"type":41,"tag":56,"props":1028,"children":1029},{"__ignoreMap":190},[1030,1057,1072,1083,1095],{"type":41,"tag":196,"props":1031,"children":1032},{"class":198,"line":199},[1033,1037,1041,1045,1049,1053],{"type":41,"tag":196,"props":1034,"children":1035},{"style":203},[1036],{"type":47,"value":23},{"type":41,"tag":196,"props":1038,"children":1039},{"style":208},[1040],{"type":47,"value":211},{"type":41,"tag":196,"props":1042,"children":1043},{"style":208},[1044],{"type":47,"value":216},{"type":41,"tag":196,"props":1046,"children":1047},{"style":208},[1048],{"type":47,"value":221},{"type":41,"tag":196,"props":1050,"children":1051},{"style":208},[1052],{"type":47,"value":226},{"type":41,"tag":196,"props":1054,"children":1055},{"style":229},[1056],{"type":47,"value":232},{"type":41,"tag":196,"props":1058,"children":1059},{"class":198,"line":235},[1060,1064,1068],{"type":41,"tag":196,"props":1061,"children":1062},{"style":208},[1063],{"type":47,"value":293},{"type":41,"tag":196,"props":1065,"children":1066},{"style":208},[1067],{"type":47,"value":298},{"type":41,"tag":196,"props":1069,"children":1070},{"style":229},[1071],{"type":47,"value":232},{"type":41,"tag":196,"props":1073,"children":1074},{"class":198,"line":253},[1075,1079],{"type":41,"tag":196,"props":1076,"children":1077},{"style":208},[1078],{"type":47,"value":311},{"type":41,"tag":196,"props":1080,"children":1081},{"style":229},[1082],{"type":47,"value":232},{"type":41,"tag":196,"props":1084,"children":1085},{"class":198,"line":287},[1086,1091],{"type":41,"tag":196,"props":1087,"children":1088},{"style":208},[1089],{"type":47,"value":1090},"  my-vllm-custom:latest",{"type":41,"tag":196,"props":1092,"children":1093},{"style":229},[1094],{"type":47,"value":232},{"type":41,"tag":196,"props":1096,"children":1097},{"class":198,"line":305},[1098,1102],{"type":41,"tag":196,"props":1099,"children":1100},{"style":208},[1101],{"type":47,"value":337},{"type":41,"tag":196,"props":1103,"children":1104},{"style":208},[1105],{"type":47,"value":342},{"type":41,"tag":441,"props":1107,"children":1109},{"id":1108},"building-for-arm64aarch64",[1110],{"type":47,"value":1111},"Building for ARM64\u002Faarch64",{"type":41,"tag":50,"props":1113,"children":1114},{},[1115,1117,1123],{"type":47,"value":1116},"A Docker container can be built for ARM64 systems (e.g., NVIDIA Grace-Hopper and Grace-Blackwell). Use the flag ",{"type":41,"tag":56,"props":1118,"children":1120},{"className":1119},[],[1121],{"type":47,"value":1122},"--platform \"linux\u002Farm64\"",{"type":47,"value":1124},":",{"type":41,"tag":185,"props":1126,"children":1128},{"className":187,"code":1127,"language":189,"meta":190,"style":190},"DOCKER_BUILDKIT=1 docker build . \\\n  --target vllm-openai \\\n  --tag vllm\u002Fvllm-openai \\\n  --file docker\u002FDockerfile \\\n  --platform \"linux\u002Farm64\"\n",[1129],{"type":41,"tag":56,"props":1130,"children":1131},{"__ignoreMap":190},[1132,1163,1178,1193,1208],{"type":41,"tag":196,"props":1133,"children":1134},{"class":198,"line":199},[1135,1139,1143,1147,1151,1155,1159],{"type":41,"tag":196,"props":1136,"children":1137},{"style":229},[1138],{"type":47,"value":461},{"type":41,"tag":196,"props":1140,"children":1141},{"style":262},[1142],{"type":47,"value":466},{"type":41,"tag":196,"props":1144,"children":1145},{"style":208},[1146],{"type":47,"value":471},{"type":41,"tag":196,"props":1148,"children":1149},{"style":203},[1150],{"type":47,"value":476},{"type":41,"tag":196,"props":1152,"children":1153},{"style":208},[1154],{"type":47,"value":481},{"type":41,"tag":196,"props":1156,"children":1157},{"style":208},[1158],{"type":47,"value":486},{"type":41,"tag":196,"props":1160,"children":1161},{"style":229},[1162],{"type":47,"value":232},{"type":41,"tag":196,"props":1164,"children":1165},{"class":198,"line":235},[1166,1170,1174],{"type":41,"tag":196,"props":1167,"children":1168},{"style":208},[1169],{"type":47,"value":498},{"type":41,"tag":196,"props":1171,"children":1172},{"style":208},[1173],{"type":47,"value":503},{"type":41,"tag":196,"props":1175,"children":1176},{"style":229},[1177],{"type":47,"value":232},{"type":41,"tag":196,"props":1179,"children":1180},{"class":198,"line":253},[1181,1185,1189],{"type":41,"tag":196,"props":1182,"children":1183},{"style":208},[1184],{"type":47,"value":515},{"type":41,"tag":196,"props":1186,"children":1187},{"style":208},[1188],{"type":47,"value":520},{"type":41,"tag":196,"props":1190,"children":1191},{"style":229},[1192],{"type":47,"value":232},{"type":41,"tag":196,"props":1194,"children":1195},{"class":198,"line":287},[1196,1200,1204],{"type":41,"tag":196,"props":1197,"children":1198},{"style":208},[1199],{"type":47,"value":532},{"type":41,"tag":196,"props":1201,"children":1202},{"style":208},[1203],{"type":47,"value":781},{"type":41,"tag":196,"props":1205,"children":1206},{"style":229},[1207],{"type":47,"value":232},{"type":41,"tag":196,"props":1209,"children":1210},{"class":198,"line":305},[1211,1216,1220,1225],{"type":41,"tag":196,"props":1212,"children":1213},{"style":208},[1214],{"type":47,"value":1215},"  --platform",{"type":41,"tag":196,"props":1217,"children":1218},{"style":262},[1219],{"type":47,"value":265},{"type":41,"tag":196,"props":1221,"children":1222},{"style":208},[1223],{"type":47,"value":1224},"linux\u002Farm64",{"type":41,"tag":196,"props":1226,"children":1227},{"style":262},[1228],{"type":47,"value":854},{"type":41,"tag":50,"props":1230,"children":1231},{},[1232,1236,1238,1244,1246,1251,1253,1259],{"type":41,"tag":396,"props":1233,"children":1234},{},[1235],{"type":47,"value":400},{"type":47,"value":1237}," Multiple modules must be compiled, so this process can take longer. Use build arguments like ",{"type":41,"tag":56,"props":1239,"children":1241},{"className":1240},[],[1242],{"type":47,"value":1243},"--build-arg max_jobs=8 --build-arg nvcc_threads=2",{"type":47,"value":1245}," to speed up the process (ensure ",{"type":41,"tag":56,"props":1247,"children":1249},{"className":1248},[],[1250],{"type":47,"value":601},{"type":47,"value":1252}," is substantially larger than ",{"type":41,"tag":56,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":47,"value":1258},"nvcc_threads",{"type":47,"value":1260},"). Monitor memory usage, as parallel jobs can require significant RAM.",{"type":41,"tag":50,"props":1262,"children":1263},{},[1264,1269],{"type":41,"tag":396,"props":1265,"children":1266},{},[1267],{"type":47,"value":1268},"For cross-compilation",{"type":47,"value":1270}," (building ARM64 on an x86_64 host), register QEMU user-static handlers first:",{"type":41,"tag":185,"props":1272,"children":1274},{"className":187,"code":1273,"language":189,"meta":190,"style":190},"docker run --rm --privileged multiarch\u002Fqemu-user-static --reset -p yes\n",[1275],{"type":41,"tag":56,"props":1276,"children":1277},{"__ignoreMap":190},[1278],{"type":41,"tag":196,"props":1279,"children":1280},{"class":198,"line":199},[1281,1285,1289,1293,1298,1303,1308,1313],{"type":41,"tag":196,"props":1282,"children":1283},{"style":203},[1284],{"type":47,"value":23},{"type":41,"tag":196,"props":1286,"children":1287},{"style":208},[1288],{"type":47,"value":211},{"type":41,"tag":196,"props":1290,"children":1291},{"style":208},[1292],{"type":47,"value":216},{"type":41,"tag":196,"props":1294,"children":1295},{"style":208},[1296],{"type":47,"value":1297}," --privileged",{"type":41,"tag":196,"props":1299,"children":1300},{"style":208},[1301],{"type":47,"value":1302}," multiarch\u002Fqemu-user-static",{"type":41,"tag":196,"props":1304,"children":1305},{"style":208},[1306],{"type":47,"value":1307}," --reset",{"type":41,"tag":196,"props":1309,"children":1310},{"style":208},[1311],{"type":47,"value":1312}," -p",{"type":41,"tag":196,"props":1314,"children":1315},{"style":208},[1316],{"type":47,"value":1317}," yes\n",{"type":41,"tag":50,"props":1319,"children":1320},{},[1321,1323,1328],{"type":47,"value":1322},"Then use the ",{"type":41,"tag":56,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":47,"value":1122},{"type":47,"value":1329}," flag in your build command.",{"type":41,"tag":441,"props":1331,"children":1333},{"id":1332},"running-your-custom-built-image",[1334],{"type":47,"value":1335},"Running your custom-built image",{"type":41,"tag":50,"props":1337,"children":1338},{},[1339],{"type":47,"value":1340},"After building, run your image just like the pre-built image:",{"type":41,"tag":185,"props":1342,"children":1344},{"className":187,"code":1343,"language":189,"meta":190,"style":190},"docker run --rm --runtime nvidia --gpus all \\\n  -v ~\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface \\\n  --env \"HF_TOKEN=$HF_TOKEN\" \\\n  -p 8000:8000 \\\n  --ipc=host \\\n  vllm\u002Fvllm-openai \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct\n",[1345],{"type":41,"tag":56,"props":1346,"children":1347},{"__ignoreMap":190},[1348,1385,1400,1427,1442,1453,1465],{"type":41,"tag":196,"props":1349,"children":1350},{"class":198,"line":199},[1351,1355,1359,1363,1368,1373,1377,1381],{"type":41,"tag":196,"props":1352,"children":1353},{"style":203},[1354],{"type":47,"value":23},{"type":41,"tag":196,"props":1356,"children":1357},{"style":208},[1358],{"type":47,"value":211},{"type":41,"tag":196,"props":1360,"children":1361},{"style":208},[1362],{"type":47,"value":216},{"type":41,"tag":196,"props":1364,"children":1365},{"style":208},[1366],{"type":47,"value":1367}," --runtime",{"type":41,"tag":196,"props":1369,"children":1370},{"style":208},[1371],{"type":47,"value":1372}," nvidia",{"type":41,"tag":196,"props":1374,"children":1375},{"style":208},[1376],{"type":47,"value":221},{"type":41,"tag":196,"props":1378,"children":1379},{"style":208},[1380],{"type":47,"value":226},{"type":41,"tag":196,"props":1382,"children":1383},{"style":229},[1384],{"type":47,"value":232},{"type":41,"tag":196,"props":1386,"children":1387},{"class":198,"line":235},[1388,1392,1396],{"type":41,"tag":196,"props":1389,"children":1390},{"style":208},[1391],{"type":47,"value":241},{"type":41,"tag":196,"props":1393,"children":1394},{"style":208},[1395],{"type":47,"value":246},{"type":41,"tag":196,"props":1397,"children":1398},{"style":229},[1399],{"type":47,"value":232},{"type":41,"tag":196,"props":1401,"children":1402},{"class":198,"line":253},[1403,1407,1411,1415,1419,1423],{"type":41,"tag":196,"props":1404,"children":1405},{"style":208},[1406],{"type":47,"value":259},{"type":41,"tag":196,"props":1408,"children":1409},{"style":262},[1410],{"type":47,"value":265},{"type":41,"tag":196,"props":1412,"children":1413},{"style":208},[1414],{"type":47,"value":270},{"type":41,"tag":196,"props":1416,"children":1417},{"style":229},[1418],{"type":47,"value":275},{"type":41,"tag":196,"props":1420,"children":1421},{"style":262},[1422],{"type":47,"value":280},{"type":41,"tag":196,"props":1424,"children":1425},{"style":229},[1426],{"type":47,"value":232},{"type":41,"tag":196,"props":1428,"children":1429},{"class":198,"line":287},[1430,1434,1438],{"type":41,"tag":196,"props":1431,"children":1432},{"style":208},[1433],{"type":47,"value":293},{"type":41,"tag":196,"props":1435,"children":1436},{"style":208},[1437],{"type":47,"value":298},{"type":41,"tag":196,"props":1439,"children":1440},{"style":229},[1441],{"type":47,"value":232},{"type":41,"tag":196,"props":1443,"children":1444},{"class":198,"line":305},[1445,1449],{"type":41,"tag":196,"props":1446,"children":1447},{"style":208},[1448],{"type":47,"value":311},{"type":41,"tag":196,"props":1450,"children":1451},{"style":229},[1452],{"type":47,"value":232},{"type":41,"tag":196,"props":1454,"children":1455},{"class":198,"line":318},[1456,1461],{"type":41,"tag":196,"props":1457,"children":1458},{"style":208},[1459],{"type":47,"value":1460},"  vllm\u002Fvllm-openai",{"type":41,"tag":196,"props":1462,"children":1463},{"style":229},[1464],{"type":47,"value":232},{"type":41,"tag":196,"props":1466,"children":1467},{"class":198,"line":331},[1468,1472],{"type":41,"tag":196,"props":1469,"children":1470},{"style":208},[1471],{"type":47,"value":337},{"type":41,"tag":196,"props":1473,"children":1474},{"style":208},[1475],{"type":47,"value":342},{"type":41,"tag":50,"props":1477,"children":1478},{},[1479,1481,1487,1489,1495],{"type":47,"value":1480},"Replace ",{"type":41,"tag":56,"props":1482,"children":1484},{"className":1483},[],[1485],{"type":47,"value":1486},"vllm\u002Fvllm-openai",{"type":47,"value":1488}," with the tag you specified during the build (e.g., ",{"type":41,"tag":56,"props":1490,"children":1492},{"className":1491},[],[1493],{"type":47,"value":1494},"my-vllm-custom:latest",{"type":47,"value":1496},").",{"type":41,"tag":389,"props":1498,"children":1499},{},[1500],{"type":41,"tag":50,"props":1501,"children":1502},{},[1503,1507,1509,1515,1517,1523,1525,1530],{"type":41,"tag":396,"props":1504,"children":1505},{},[1506],{"type":47,"value":400},{"type":47,"value":1508}," ",{"type":41,"tag":56,"props":1510,"children":1512},{"className":1511},[],[1513],{"type":47,"value":1514},"--runtime nvidia",{"type":47,"value":1516}," is deprecated for most environments. Prefer ",{"type":41,"tag":56,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":47,"value":1522},"--gpus ...",{"type":47,"value":1524}," with NVIDIA Container Toolkit. Use ",{"type":41,"tag":56,"props":1526,"children":1528},{"className":1527},[],[1529],{"type":47,"value":1514},{"type":47,"value":1531}," only for legacy Docker configurations.",{"type":41,"tag":84,"props":1533,"children":1535},{"id":1534},"common-server-flags",[1536],{"type":47,"value":1537},"Common server flags",{"type":41,"tag":91,"props":1539,"children":1540},{},[1541,1552,1563,1574],{"type":41,"tag":95,"props":1542,"children":1543},{},[1544,1550],{"type":41,"tag":56,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":47,"value":1549},"--model \u003CMODEL_ID>",{"type":47,"value":1551}," — model to load (HF ID or local path)",{"type":41,"tag":95,"props":1553,"children":1554},{},[1555,1561],{"type":41,"tag":56,"props":1556,"children":1558},{"className":1557},[],[1559],{"type":47,"value":1560},"--port \u003CPORT>",{"type":47,"value":1562}," — server port (default 8000 for OpenAI-compatible server)",{"type":41,"tag":95,"props":1564,"children":1565},{},[1566,1572],{"type":41,"tag":56,"props":1567,"children":1569},{"className":1568},[],[1570],{"type":47,"value":1571},"--log-level",{"type":47,"value":1573}," — adjust verbosity",{"type":41,"tag":95,"props":1575,"children":1576},{},[1577,1579,1585],{"type":47,"value":1578},"You may pass additional ",{"type":41,"tag":56,"props":1580,"children":1582},{"className":1581},[],[1583],{"type":47,"value":1584},"engine_args",{"type":47,"value":1586}," after the image tag; see vLLM docs for tuning options.",{"type":41,"tag":84,"props":1588,"children":1590},{"id":1589},"testing-the-api",[1591],{"type":47,"value":1592},"Testing the API",{"type":41,"tag":50,"props":1594,"children":1595},{},[1596],{"type":47,"value":1597},"After the container starts, make a quick test request against the OpenAI-compatible endpoint:",{"type":41,"tag":185,"props":1599,"children":1601},{"className":187,"code":1600,"language":189,"meta":190,"style":190},"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"model\":\"Qwen\u002FQwen2.5-1.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Who are you?\"}],\"max_tokens\":128}'\n",[1602],{"type":41,"tag":56,"props":1603,"children":1604},{"__ignoreMap":190},[1605,1626,1651],{"type":41,"tag":196,"props":1606,"children":1607},{"class":198,"line":199},[1608,1612,1617,1622],{"type":41,"tag":196,"props":1609,"children":1610},{"style":203},[1611],{"type":47,"value":160},{"type":41,"tag":196,"props":1613,"children":1614},{"style":208},[1615],{"type":47,"value":1616}," -s",{"type":41,"tag":196,"props":1618,"children":1619},{"style":208},[1620],{"type":47,"value":1621}," http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions",{"type":41,"tag":196,"props":1623,"children":1624},{"style":229},[1625],{"type":47,"value":232},{"type":41,"tag":196,"props":1627,"children":1628},{"class":198,"line":235},[1629,1634,1638,1643,1647],{"type":41,"tag":196,"props":1630,"children":1631},{"style":208},[1632],{"type":47,"value":1633},"  -H",{"type":41,"tag":196,"props":1635,"children":1636},{"style":262},[1637],{"type":47,"value":265},{"type":41,"tag":196,"props":1639,"children":1640},{"style":208},[1641],{"type":47,"value":1642},"Content-Type: application\u002Fjson",{"type":41,"tag":196,"props":1644,"children":1645},{"style":262},[1646],{"type":47,"value":280},{"type":41,"tag":196,"props":1648,"children":1649},{"style":229},[1650],{"type":47,"value":232},{"type":41,"tag":196,"props":1652,"children":1653},{"class":198,"line":253},[1654,1659,1664,1669],{"type":41,"tag":196,"props":1655,"children":1656},{"style":208},[1657],{"type":47,"value":1658},"  -d",{"type":41,"tag":196,"props":1660,"children":1661},{"style":262},[1662],{"type":47,"value":1663}," '",{"type":41,"tag":196,"props":1665,"children":1666},{"style":208},[1667],{"type":47,"value":1668},"{\"model\":\"Qwen\u002FQwen2.5-1.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"Who are you?\"}],\"max_tokens\":128}",{"type":41,"tag":196,"props":1670,"children":1671},{"style":262},[1672],{"type":47,"value":1673},"'\n",{"type":41,"tag":84,"props":1675,"children":1677},{"id":1676},"security-and-operational-notes",[1678],{"type":47,"value":1679},"Security and operational notes",{"type":41,"tag":91,"props":1681,"children":1682},{},[1683,1695,1700],{"type":41,"tag":95,"props":1684,"children":1685},{},[1686,1688,1693],{"type":47,"value":1687},"Keep ",{"type":41,"tag":56,"props":1689,"children":1691},{"className":1690},[],[1692],{"type":47,"value":128},{"type":47,"value":1694}," secret; prefer passing it via environment variables or a secret manager.",{"type":41,"tag":95,"props":1696,"children":1697},{},[1698],{"type":47,"value":1699},"For production, run behind a reverse proxy (Nginx) with TLS and authentication.",{"type":41,"tag":95,"props":1701,"children":1702},{},[1703],{"type":47,"value":1704},"Mount only necessary host paths into the container.",{"type":41,"tag":84,"props":1706,"children":1708},{"id":1707},"troubleshooting",[1709],{"type":47,"value":1710},"Troubleshooting",{"type":41,"tag":91,"props":1712,"children":1713},{},[1714,1727,1739,1750,1763],{"type":41,"tag":95,"props":1715,"children":1716},{},[1717,1719,1725],{"type":47,"value":1718},"Container can't access GPUs: ensure ",{"type":41,"tag":56,"props":1720,"children":1722},{"className":1721},[],[1723],{"type":47,"value":1724},"nvidia-container-toolkit",{"type":47,"value":1726}," is installed and restart Docker.",{"type":41,"tag":95,"props":1728,"children":1729},{},[1730,1732,1737],{"type":47,"value":1731},"Model download failures: check ",{"type":41,"tag":56,"props":1733,"children":1735},{"className":1734},[],[1736],{"type":47,"value":128},{"type":47,"value":1738}," and network; mount cache directory to persist downloads.",{"type":41,"tag":95,"props":1740,"children":1741},{},[1742,1744,1749],{"type":47,"value":1743},"Memory \u002F OOM errors: try a smaller model or add more GPU memory; check ",{"type":41,"tag":56,"props":1745,"children":1747},{"className":1746},[],[1748],{"type":47,"value":372},{"type":47,"value":419},{"type":41,"tag":95,"props":1751,"children":1752},{},[1753,1755,1761],{"type":47,"value":1754},"If the container fails with NCCL library path issues (rare): set ",{"type":41,"tag":56,"props":1756,"children":1758},{"className":1757},[],[1759],{"type":47,"value":1760},"VLLM_NCCL_SO_PATH",{"type":47,"value":1762}," per upstream guidance.",{"type":41,"tag":95,"props":1764,"children":1765},{},[1766,1768,1773,1775,1780],{"type":47,"value":1767},"Permission issues of current user with Docker: ensure your user is in the ",{"type":41,"tag":56,"props":1769,"children":1771},{"className":1770},[],[1772],{"type":47,"value":23},{"type":47,"value":1774}," group or suggest user to add current user to ",{"type":41,"tag":56,"props":1776,"children":1778},{"className":1777},[],[1779],{"type":47,"value":23},{"type":47,"value":1781}," group manually following:",{"type":41,"tag":185,"props":1783,"children":1785},{"className":187,"code":1784,"language":189,"meta":190,"style":190},"# 1. Create docker group if it doesn't exist (may already exist on some systems)\nsudo groupadd docker\n\n# 2. Add current user to the docker group (replace $USER with your username if needed)\nsudo usermod -aG docker $USER\n\n# 3. Apply the new group membership (you may need to log out and log back in for this to take effect)\nnewgrp docker\n\n# 4. Verify that the user is in the docker group (output should include docker)\ngroups $USER\n",[1786],{"type":41,"tag":56,"props":1787,"children":1788},{"__ignoreMap":190},[1789,1798,1816,1823,1831,1857,1864,1872,1885,1893,1902],{"type":41,"tag":196,"props":1790,"children":1791},{"class":198,"line":199},[1792],{"type":41,"tag":196,"props":1793,"children":1795},{"style":1794},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1796],{"type":47,"value":1797},"# 1. Create docker group if it doesn't exist (may already exist on some systems)\n",{"type":41,"tag":196,"props":1799,"children":1800},{"class":198,"line":235},[1801,1806,1811],{"type":41,"tag":196,"props":1802,"children":1803},{"style":203},[1804],{"type":47,"value":1805},"sudo",{"type":41,"tag":196,"props":1807,"children":1808},{"style":208},[1809],{"type":47,"value":1810}," groupadd",{"type":41,"tag":196,"props":1812,"children":1813},{"style":208},[1814],{"type":47,"value":1815}," docker\n",{"type":41,"tag":196,"props":1817,"children":1818},{"class":198,"line":253},[1819],{"type":41,"tag":196,"props":1820,"children":1821},{"emptyLinePlaceholder":904},[1822],{"type":47,"value":907},{"type":41,"tag":196,"props":1824,"children":1825},{"class":198,"line":287},[1826],{"type":41,"tag":196,"props":1827,"children":1828},{"style":1794},[1829],{"type":47,"value":1830},"# 2. Add current user to the docker group (replace $USER with your username if needed)\n",{"type":41,"tag":196,"props":1832,"children":1833},{"class":198,"line":305},[1834,1838,1843,1848,1852],{"type":41,"tag":196,"props":1835,"children":1836},{"style":203},[1837],{"type":47,"value":1805},{"type":41,"tag":196,"props":1839,"children":1840},{"style":208},[1841],{"type":47,"value":1842}," usermod",{"type":41,"tag":196,"props":1844,"children":1845},{"style":208},[1846],{"type":47,"value":1847}," -aG",{"type":41,"tag":196,"props":1849,"children":1850},{"style":208},[1851],{"type":47,"value":476},{"type":41,"tag":196,"props":1853,"children":1854},{"style":229},[1855],{"type":47,"value":1856}," $USER\n",{"type":41,"tag":196,"props":1858,"children":1859},{"class":198,"line":318},[1860],{"type":41,"tag":196,"props":1861,"children":1862},{"emptyLinePlaceholder":904},[1863],{"type":47,"value":907},{"type":41,"tag":196,"props":1865,"children":1866},{"class":198,"line":331},[1867],{"type":41,"tag":196,"props":1868,"children":1869},{"style":1794},[1870],{"type":47,"value":1871},"# 3. Apply the new group membership (you may need to log out and log back in for this to take effect)\n",{"type":41,"tag":196,"props":1873,"children":1875},{"class":198,"line":1874},8,[1876,1881],{"type":41,"tag":196,"props":1877,"children":1878},{"style":203},[1879],{"type":47,"value":1880},"newgrp",{"type":41,"tag":196,"props":1882,"children":1883},{"style":208},[1884],{"type":47,"value":1815},{"type":41,"tag":196,"props":1886,"children":1888},{"class":198,"line":1887},9,[1889],{"type":41,"tag":196,"props":1890,"children":1891},{"emptyLinePlaceholder":904},[1892],{"type":47,"value":907},{"type":41,"tag":196,"props":1894,"children":1896},{"class":198,"line":1895},10,[1897],{"type":41,"tag":196,"props":1898,"children":1899},{"style":1794},[1900],{"type":47,"value":1901},"# 4. Verify that the user is in the docker group (output should include docker)\n",{"type":41,"tag":196,"props":1903,"children":1905},{"class":198,"line":1904},11,[1906,1911],{"type":41,"tag":196,"props":1907,"children":1908},{"style":203},[1909],{"type":47,"value":1910},"groups",{"type":41,"tag":196,"props":1912,"children":1913},{"style":229},[1914],{"type":47,"value":1856},{"type":41,"tag":91,"props":1916,"children":1917},{},[1918,1923],{"type":41,"tag":95,"props":1919,"children":1920},{},[1921],{"type":47,"value":1922},"Docker pull failed due to a network timeout connecting to Docker Hub: check the network connection and try again. If the issue persists, suggest and show the user how to use a mirror for Docker Hub.",{"type":41,"tag":95,"props":1924,"children":1925},{},[1926,1928,1933,1935,1941,1943,1949],{"type":47,"value":1927},"Error during downloading models from Hugging Face: check if the ",{"type":41,"tag":56,"props":1929,"children":1931},{"className":1930},[],[1932],{"type":47,"value":128},{"type":47,"value":1934}," is passed to the container and is valid. Check if ",{"type":41,"tag":56,"props":1936,"children":1938},{"className":1937},[],[1939],{"type":47,"value":1940},"HTTP_PROXY",{"type":47,"value":1942}," and ",{"type":41,"tag":56,"props":1944,"children":1946},{"className":1945},[],[1947],{"type":47,"value":1948},"HTTPS_PROXY",{"type":47,"value":1950}," are passed to the container if the host is behind a proxy. Also, verify that the model ID is correct and that the model is public or accessible with the provided token.",{"type":41,"tag":84,"props":1952,"children":1954},{"id":1953},"references",[1955],{"type":47,"value":1956},"References",{"type":41,"tag":91,"props":1958,"children":1959},{},[1960,1971,1982],{"type":41,"tag":95,"props":1961,"children":1962},{},[1963,1965],{"type":47,"value":1964},"vLLM repository (docker\u002FDockerfile): ",{"type":41,"tag":73,"props":1966,"children":1969},{"href":1967,"rel":1968},"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Ftree\u002Fmain\u002Fdocker",[77],[1970],{"type":47,"value":1967},{"type":41,"tag":95,"props":1972,"children":1973},{},[1974,1976],{"type":47,"value":1975},"NVIDIA Container Toolkit: ",{"type":41,"tag":73,"props":1977,"children":1980},{"href":1978,"rel":1979},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fcontainer-toolkit\u002Flatest\u002Finstall-guide.html",[77],[1981],{"type":47,"value":1978},{"type":41,"tag":95,"props":1983,"children":1984},{},[1985,1987],{"type":47,"value":1986},"Up-to-date deployment instructions and troubleshooting: ",{"type":41,"tag":73,"props":1988,"children":1991},{"href":1989,"rel":1990},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fdeployment\u002Fdocker\u002F",[77],[1992],{"type":47,"value":1989},{"type":41,"tag":1994,"props":1995,"children":1996},"style",{},[1997],{"type":47,"value":1998},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2000,"total":331},[2001,2018,2033,2047,2054,2070,2081],{"slug":2002,"name":2002,"fn":2003,"description":2004,"org":2005,"tags":2006,"stars":2015,"repoUrl":2016,"updatedAt":2017},"openclaw-vsr-bridge","integrate vLLM Semantic Router","Install vLLM Semantic Router in agent-safe mode, import supported OpenClaw model providers into canonical VSR config, and rewrite OpenClaw to target VSR.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2007,2010,2011,2014],{"name":2008,"slug":2009,"type":16},"Agents","agents",{"name":14,"slug":15,"type":16},{"name":2012,"slug":2013,"type":16},"Routing","routing",{"name":9,"slug":8,"type":16},4962,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fsemantic-router","2026-07-18T05:48:09.091161",{"slug":2019,"name":2019,"fn":2020,"description":2021,"org":2022,"tags":2023,"stars":24,"repoUrl":25,"updatedAt":2032},"vllm-bench-random-synthetic","run vLLM synthetic performance benchmarks","Run vLLM performance benchmark using synthetic random data to measure throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and other key performance metrics. Use when the user wants to quickly test vLLM serving performance without downloading external datasets.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2024,2027,2028,2031],{"name":2025,"slug":2026,"type":16},"Benchmarking","benchmarking",{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},"Performance","performance",{"name":9,"slug":8,"type":16},"2026-07-18T05:46:59.036927",{"slug":2034,"name":2034,"fn":2035,"description":2036,"org":2037,"tags":2038,"stars":24,"repoUrl":25,"updatedAt":2046},"vllm-bench-serve","benchmark LLM serving endpoints","Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve. Supports multiple datasets (random, sharegpt, sonnet, HF), backends (openai, openai-chat, vllm-pooling, embeddings), throughput\u002Flatency testing with request-rate control, and result saving. Use when benchmarking LLM serving performance, measuring TTFT\u002FTPOT, or load testing inference APIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2039,2042,2043,2044,2045],{"name":2040,"slug":2041,"type":16},"API Development","api-development",{"name":2025,"slug":2026,"type":16},{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:02.283164",{"slug":4,"name":4,"fn":5,"description":6,"org":2048,"tags":2049,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2050,2051,2052,2053],{"name":19,"slug":20,"type":16},{"name":22,"slug":23,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2055,"name":2055,"fn":2056,"description":2057,"org":2058,"tags":2059,"stars":24,"repoUrl":25,"updatedAt":2069},"vllm-deploy-k8s","deploy vLLM to Kubernetes clusters","Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint. Use this skill whenever the user wants to deploy, run, or serve vLLM on a Kubernetes cluster, including creating deployments, services, checking existing deployments, or managing vLLM on K8s.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2060,2063,2064,2067,2068],{"name":2061,"slug":2062,"type":16},"AI Infrastructure","ai-infrastructure",{"name":19,"slug":20,"type":16},{"name":2065,"slug":2066,"type":16},"Kubernetes","kubernetes",{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:46:58.556637",{"slug":2071,"name":2071,"fn":2072,"description":2073,"org":2074,"tags":2075,"stars":24,"repoUrl":25,"updatedAt":2080},"vllm-deploy-simple","deploy and test vLLM servers","Quick install and deploy vLLM, start serving with a simple LLM, and test OpenAI API.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2076,2077,2078,2079],{"name":2040,"slug":2041,"type":16},{"name":19,"slug":20,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:01.359655",{"slug":2082,"name":2082,"fn":2083,"description":2084,"org":2085,"tags":2086,"stars":24,"repoUrl":25,"updatedAt":2091},"vllm-prefix-cache-bench","benchmark vLLM prefix caching efficiency","This is a skill for benchmarking the efficiency of automatic prefix caching in vLLM using fixed prompts, real-world datasets, or synthetic prefix\u002Fsuffix patterns. Use when the user asks to benchmark prefix caching hit rate, caching efficiency, or repeated-prompt performance in vLLM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2087,2088,2089,2090],{"name":2025,"slug":2026,"type":16},{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:48:07.243092",{"items":2093,"total":318},[2094,2101,2109,2116,2124,2131],{"slug":2019,"name":2019,"fn":2020,"description":2021,"org":2095,"tags":2096,"stars":24,"repoUrl":25,"updatedAt":2032},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2097,2098,2099,2100],{"name":2025,"slug":2026,"type":16},{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},{"name":9,"slug":8,"type":16},{"slug":2034,"name":2034,"fn":2035,"description":2036,"org":2102,"tags":2103,"stars":24,"repoUrl":25,"updatedAt":2046},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2104,2105,2106,2107,2108],{"name":2040,"slug":2041,"type":16},{"name":2025,"slug":2026,"type":16},{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},{"name":9,"slug":8,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":2110,"tags":2111,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2112,2113,2114,2115],{"name":19,"slug":20,"type":16},{"name":22,"slug":23,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2055,"name":2055,"fn":2056,"description":2057,"org":2117,"tags":2118,"stars":24,"repoUrl":25,"updatedAt":2069},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2119,2120,2121,2122,2123],{"name":2061,"slug":2062,"type":16},{"name":19,"slug":20,"type":16},{"name":2065,"slug":2066,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2071,"name":2071,"fn":2072,"description":2073,"org":2125,"tags":2126,"stars":24,"repoUrl":25,"updatedAt":2080},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2127,2128,2129,2130],{"name":2040,"slug":2041,"type":16},{"name":19,"slug":20,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2082,"name":2082,"fn":2083,"description":2084,"org":2132,"tags":2133,"stars":24,"repoUrl":25,"updatedAt":2091},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2134,2135,2136,2137],{"name":2025,"slug":2026,"type":16},{"name":14,"slug":15,"type":16},{"name":2029,"slug":2030,"type":16},{"name":9,"slug":8,"type":16}]