[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-vllm-vllm-deploy-k8s":3,"mdc--ht60g6-key":38,"related-org-vllm-vllm-deploy-k8s":1738,"related-repo-vllm-vllm-deploy-k8s":1830},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"vllm-deploy-k8s","deploy vLLM to Kubernetes clusters","Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint. Use this skill whenever the user wants to deploy, run, or serve vLLM on a Kubernetes cluster, including creating deployments, services, checking existing deployments, or managing vLLM on K8s.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"vllm","vLLM","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fvllm.png","vllm-project",[13,17,18,21,24],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":9,"slug":8,"type":16},{"name":19,"slug":20,"type":16},"Deployment","deployment",{"name":22,"slug":23,"type":16},"AI Infrastructure","ai-infrastructure",{"name":25,"slug":26,"type":16},"Kubernetes","kubernetes",88,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills","2026-07-18T05:46:58.556637",null,23,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Agent skills for vLLM","https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fvllm-skills\u002Fskills\u002Fvllm-deploy-k8s","---\nname: vllm-deploy-k8s\ndescription: Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint. Use this skill whenever the user wants to deploy, run, or serve vLLM on a Kubernetes cluster, including creating deployments, services, checking existing deployments, or managing vLLM on K8s.\n---\n\n# vLLM Kubernetes Deployment\n\nA Claude skill for deploying vLLM to Kubernetes using YAML templates. Deploys a vLLM OpenAI-compatible server as a Kubernetes Deployment with a ClusterIP Service, GPU resources, and health probes.\n\n## What this skill does\n\n- Deploy vLLM as a Kubernetes Deployment + Service with NVIDIA GPU support\n- Check if a vLLM deployment already exists before deploying\n- Check if the Hugging Face token secret exists, and ask the user for their token if not\n- Use the `vllm\u002Fvllm-openai:latest` image by default (user can specify a different version)\n- Provide sensible default configuration that users can customize (model, replicas, GPU count, extra vLLM flags, etc.)\n\n## Prerequisites\n\n- `kubectl` configured with access to a Kubernetes cluster\n- NVIDIA GPU Operator or device plugin installed on cluster nodes\n- Hugging Face token (required for gated models like Llama, optional for public models)\n\n## Deployment Steps\n\n### Step 1: Check HF token secret\n\nBefore deploying, check if the `hf-token` Kubernetes secret exists in the target namespace:\n\n```bash\nkubectl get secret hf-token -n \u003Cnamespace>\n```\n\n- If the secret **exists**: proceed to Step 2.\n- If the secret **does not exist**: ask the user to provide their Hugging Face token, then create the secret:\n\n```bash\nkubectl create secret generic hf-token --from-literal=HF_TOKEN=\"\u003Cuser-provided-token>\" -n \u003Cnamespace>\n```\n\nThis is required for gated models (e.g., `meta-llama\u002FMeta-Llama-3.1-8B`). For public models, the secret is optional but recommended to avoid rate limits.\n\n### Step 2: Check if deployment already exists\n\nBefore applying, check if a vLLM deployment already exists:\n\n```bash\nkubectl get deployment vllm -n \u003Cnamespace>\n```\n\n- If it **exists**: inform the user that the deployment already exists. Show the current image and status. Ask the user if they want to update it or skip.\n- If it **does not exist**: proceed to deploy.\n\n### Step 3: Deploy\n\nApply the template YAML files to deploy vLLM:\n\n```bash\nkubectl apply -f templates\u002Fvllm-service.yaml -n \u003Cnamespace>\nkubectl apply -f templates\u002Fvllm-deployment.yaml -n \u003Cnamespace>\n```\n\n### Step 4: Wait and verify\n\nWait for the deployment to roll out:\n\n```bash\nkubectl rollout status deployment\u002Fvllm -n \u003Cnamespace> --timeout=600s\n```\n\nVerify the pod is running and ready:\n\n```bash\nkubectl get pods -n \u003Cnamespace> -l app=vllm\n```\n\nConfirm the pod shows `READY 1\u002F1` and `STATUS Running`. If the pod is not ready yet, wait and check again. If it's in `CrashLoopBackOff` or `Error`, check the logs with `kubectl logs -n \u003Cnamespace> -l app=vllm`.\n\n### Step 5: Print deployment summary\n\nOnce the pod is ready, print a summary message to the user in this format (replace placeholders with actual values):\n\n```\n🎉 **vLLM Deployment Successful!**\n\n| Resource | Name | Status |\n|----------|------|--------|\n| Deployment | \u003Cdeployment-name> | \u003Cready>\u002F\u003Ctotal> Ready |\n| Service | \u003Cservice-name> | ClusterIP:\u003Cport> |\n| Pod | \u003Cpod-name> | Running |\n| Image | \u003Cimage> | |\n| Model | \u003Cmodel> | |\n\n \n\n**To test the API, run these two commands in your terminal:**\n\n**1. Open a port-forward** (this connects your local port \u003Cport> to the vLLM service inside the cluster):\n\nkubectl port-forward svc\u002Fvllm-svc \u003Cport>:\u003Cport> -n \u003Cnamespace>\n\n**2. In a separate terminal**, send a test request to the OpenAI-compatible API:\n\ncurl -s http:\u002F\u002Flocalhost:\u003Cport>\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"model\":\"\u003Cmodel>\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}],\"max_tokens\":50}' | python3 -m json.tool\n\nIf everything is working, you'll get a JSON response with the model's reply.\n```\n\n## Default Configuration\n\nThe templates use the following defaults:\n\n| Parameter | Default Value |\n|-----------|---------------|\n| Image | `vllm\u002Fvllm-openai:latest` |\n| Model | `Qwen\u002FQwen2.5-1.5B-Instruct` |\n| Port | `8000` |\n| Replicas | `1` |\n| GPU count | `1` |\n| GPU memory utilization | `0.85` |\n| Tensor parallel size | `1` |\n| CPU request \u002F limit | `12` \u002F `128` |\n| Memory request \u002F limit | `100Gi` \u002F `400Gi` |\n| Shared memory (dshm) | `80Gi` |\n\n## Customization\n\nWhen the user requests changes, modify the template YAML files before applying. The following can be customized:\n\n- **Image version**: Change `image: vllm\u002Fvllm-openai:\u003Cversion>` in `templates\u002Fvllm-deployment.yaml` (default: `latest`). Use a specific version tag like `v0.17.1` if the user requests it.\n- **Model**: Change the model name in the `vllm serve` command inside the Deployment `args`.\n- **Extra vLLM flags**: Append additional flags to the `vllm serve` command in the Deployment `args` (e.g., `--max-model-len 4096`, `--kv-cache-dtype fp8`, `--enforce-eager`, `--generation-config vllm`).\n- **Replicas**: Change `replicas:` in the Deployment spec.\n- **GPU count**: Change `nvidia.com\u002Fgpu` in both `requests` and `limits` under resources.\n- **Tensor parallel size**: Change `--tensor-parallel-size` flag to match the GPU count.\n- **CPU\u002FMemory resources**: Change `cpu` and `memory` values under `requests` and `limits`.\n- **Port**: Change `containerPort` in the Deployment, `port`\u002F`targetPort` in the Service, the `port` in all health probes (liveness, readiness, startup), AND add `--port \u003Cport>` to the `vllm serve` command in args. All four must match.\n- **Namespace**: Apply to a specific namespace using `-n \u003Cnamespace>`.\n- **Shared memory size**: Change the `sizeLimit` of the `dshm` emptyDir volume.\n\nEdit the template files using the Edit tool, then apply the modified templates.\n\n## Status Check\n\n```bash\nkubectl get deployment,svc,pods -n \u003Cnamespace> -l app=vllm\n```\n\n## Cleanup\n\nWhen the user asks to clean up or delete the vLLM deployment, run the following steps:\n\n1. Delete the Deployment and Service:\n\n```bash\nkubectl delete -f templates\u002Fvllm-deployment.yaml -n \u003Cnamespace>\nkubectl delete -f templates\u002Fvllm-service.yaml -n \u003Cnamespace>\n```\n\n2. Ask the user if they also want to delete the HF token secret. If yes:\n\n```bash\nkubectl delete secret hf-token -n \u003Cnamespace>\n```\n\n3. Verify everything is cleaned up:\n\n```bash\nkubectl get deployment,svc,pods -n \u003Cnamespace> -l app=vllm\n```\n\n4. Print a summary message to the user:\n\n```\nvLLM deployment has been cleaned up from namespace \u003Cnamespace>.\nDeleted: Deployment\u002Fvllm, Service\u002Fvllm-svc\nHF token secret: \u003Ckept\u002Fdeleted>\n```\n\n## Troubleshooting\n\n- **Pod stuck in Pending**: No GPU nodes available. Check `kubectl describe pod \u003Cpod-name>` for scheduling errors. Ensure NVIDIA GPU Operator or device plugin is installed.\n- **Pod OOMKilled**: Increase `memory` limits in the Deployment, or use a smaller model.\n- **ImagePullBackOff**: Check the image name and tag. Verify the node has access to Docker Hub \u002F the container registry.\n- **Startup probe failures (CrashLoopBackOff)**: Model download may be slow. Check logs with `kubectl logs \u003Cpod-name>`. Ensure `hf-token` secret exists for gated models. Increase `failureThreshold` on the startup probe if needed.\n- **HF_TOKEN not working**: Verify the secret exists: `kubectl get secret hf-token -n \u003Cnamespace>`. Check the token is valid.\n- **GPU not detected in container**: Ensure `nvidia.com\u002Fgpu` resource is requested and the NVIDIA device plugin is running on the node.\n\n## References\n\n- [vLLM Documentation](https:\u002F\u002Fdocs.vllm.ai\u002F)\n- [vLLM Docker Images](https:\u002F\u002Fhub.docker.com\u002Fr\u002Fvllm\u002Fvllm-openai\u002Ftags)\n- [NVIDIA GPU Operator](https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Findex.html)\n- [Kubernetes GPU Scheduling](https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fmanage-gpus\u002Fscheduling-gpus\u002F)\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,58,65,104,110,134,140,147,160,225,252,323,336,342,347,395,419,425,430,520,526,531,586,591,644,689,695,700,710,716,721,931,937,942,1261,1266,1272,1323,1329,1334,1343,1429,1437,1483,1492,1541,1550,1559,1565,1681,1687,1732],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"vllm-kubernetes-deployment",[49],{"type":50,"value":51},"text","vLLM Kubernetes Deployment",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56],{"type":50,"value":57},"A Claude skill for deploying vLLM to Kubernetes using YAML templates. Deploys a vLLM OpenAI-compatible server as a Kubernetes Deployment with a ClusterIP Service, GPU resources, and health probes.",{"type":44,"tag":59,"props":60,"children":62},"h2",{"id":61},"what-this-skill-does",[63],{"type":50,"value":64},"What this skill does",{"type":44,"tag":66,"props":67,"children":68},"ul",{},[69,75,80,85,99],{"type":44,"tag":70,"props":71,"children":72},"li",{},[73],{"type":50,"value":74},"Deploy vLLM as a Kubernetes Deployment + Service with NVIDIA GPU support",{"type":44,"tag":70,"props":76,"children":77},{},[78],{"type":50,"value":79},"Check if a vLLM deployment already exists before deploying",{"type":44,"tag":70,"props":81,"children":82},{},[83],{"type":50,"value":84},"Check if the Hugging Face token secret exists, and ask the user for their token if not",{"type":44,"tag":70,"props":86,"children":87},{},[88,90,97],{"type":50,"value":89},"Use the ",{"type":44,"tag":91,"props":92,"children":94},"code",{"className":93},[],[95],{"type":50,"value":96},"vllm\u002Fvllm-openai:latest",{"type":50,"value":98}," image by default (user can specify a different version)",{"type":44,"tag":70,"props":100,"children":101},{},[102],{"type":50,"value":103},"Provide sensible default configuration that users can customize (model, replicas, GPU count, extra vLLM flags, etc.)",{"type":44,"tag":59,"props":105,"children":107},{"id":106},"prerequisites",[108],{"type":50,"value":109},"Prerequisites",{"type":44,"tag":66,"props":111,"children":112},{},[113,124,129],{"type":44,"tag":70,"props":114,"children":115},{},[116,122],{"type":44,"tag":91,"props":117,"children":119},{"className":118},[],[120],{"type":50,"value":121},"kubectl",{"type":50,"value":123}," configured with access to a Kubernetes cluster",{"type":44,"tag":70,"props":125,"children":126},{},[127],{"type":50,"value":128},"NVIDIA GPU Operator or device plugin installed on cluster nodes",{"type":44,"tag":70,"props":130,"children":131},{},[132],{"type":50,"value":133},"Hugging Face token (required for gated models like Llama, optional for public models)",{"type":44,"tag":59,"props":135,"children":137},{"id":136},"deployment-steps",[138],{"type":50,"value":139},"Deployment Steps",{"type":44,"tag":141,"props":142,"children":144},"h3",{"id":143},"step-1-check-hf-token-secret",[145],{"type":50,"value":146},"Step 1: Check HF token secret",{"type":44,"tag":53,"props":148,"children":149},{},[150,152,158],{"type":50,"value":151},"Before deploying, check if the ",{"type":44,"tag":91,"props":153,"children":155},{"className":154},[],[156],{"type":50,"value":157},"hf-token",{"type":50,"value":159}," Kubernetes secret exists in the target namespace:",{"type":44,"tag":161,"props":162,"children":167},"pre",{"className":163,"code":164,"language":165,"meta":166,"style":166},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","kubectl get secret hf-token -n \u003Cnamespace>\n","bash","",[168],{"type":44,"tag":91,"props":169,"children":170},{"__ignoreMap":166},[171],{"type":44,"tag":172,"props":173,"children":176},"span",{"class":174,"line":175},"line",1,[177,182,188,193,198,203,209,214,220],{"type":44,"tag":172,"props":178,"children":180},{"style":179},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[181],{"type":50,"value":121},{"type":44,"tag":172,"props":183,"children":185},{"style":184},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[186],{"type":50,"value":187}," get",{"type":44,"tag":172,"props":189,"children":190},{"style":184},[191],{"type":50,"value":192}," secret",{"type":44,"tag":172,"props":194,"children":195},{"style":184},[196],{"type":50,"value":197}," hf-token",{"type":44,"tag":172,"props":199,"children":200},{"style":184},[201],{"type":50,"value":202}," -n",{"type":44,"tag":172,"props":204,"children":206},{"style":205},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[207],{"type":50,"value":208}," \u003C",{"type":44,"tag":172,"props":210,"children":211},{"style":184},[212],{"type":50,"value":213},"namespac",{"type":44,"tag":172,"props":215,"children":217},{"style":216},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[218],{"type":50,"value":219},"e",{"type":44,"tag":172,"props":221,"children":222},{"style":205},[223],{"type":50,"value":224},">\n",{"type":44,"tag":66,"props":226,"children":227},{},[228,241],{"type":44,"tag":70,"props":229,"children":230},{},[231,233,239],{"type":50,"value":232},"If the secret ",{"type":44,"tag":234,"props":235,"children":236},"strong",{},[237],{"type":50,"value":238},"exists",{"type":50,"value":240},": proceed to Step 2.",{"type":44,"tag":70,"props":242,"children":243},{},[244,245,250],{"type":50,"value":232},{"type":44,"tag":234,"props":246,"children":247},{},[248],{"type":50,"value":249},"does not exist",{"type":50,"value":251},": ask the user to provide their Hugging Face token, then create the secret:",{"type":44,"tag":161,"props":253,"children":255},{"className":163,"code":254,"language":165,"meta":166,"style":166},"kubectl create secret generic hf-token --from-literal=HF_TOKEN=\"\u003Cuser-provided-token>\" -n \u003Cnamespace>\n",[256],{"type":44,"tag":91,"props":257,"children":258},{"__ignoreMap":166},[259],{"type":44,"tag":172,"props":260,"children":261},{"class":174,"line":175},[262,266,271,275,280,284,289,294,299,303,307,311,315,319],{"type":44,"tag":172,"props":263,"children":264},{"style":179},[265],{"type":50,"value":121},{"type":44,"tag":172,"props":267,"children":268},{"style":184},[269],{"type":50,"value":270}," create",{"type":44,"tag":172,"props":272,"children":273},{"style":184},[274],{"type":50,"value":192},{"type":44,"tag":172,"props":276,"children":277},{"style":184},[278],{"type":50,"value":279}," generic",{"type":44,"tag":172,"props":281,"children":282},{"style":184},[283],{"type":50,"value":197},{"type":44,"tag":172,"props":285,"children":286},{"style":184},[287],{"type":50,"value":288}," --from-literal=HF_TOKEN=",{"type":44,"tag":172,"props":290,"children":291},{"style":205},[292],{"type":50,"value":293},"\"",{"type":44,"tag":172,"props":295,"children":296},{"style":184},[297],{"type":50,"value":298},"\u003Cuser-provided-token>",{"type":44,"tag":172,"props":300,"children":301},{"style":205},[302],{"type":50,"value":293},{"type":44,"tag":172,"props":304,"children":305},{"style":184},[306],{"type":50,"value":202},{"type":44,"tag":172,"props":308,"children":309},{"style":205},[310],{"type":50,"value":208},{"type":44,"tag":172,"props":312,"children":313},{"style":184},[314],{"type":50,"value":213},{"type":44,"tag":172,"props":316,"children":317},{"style":216},[318],{"type":50,"value":219},{"type":44,"tag":172,"props":320,"children":321},{"style":205},[322],{"type":50,"value":224},{"type":44,"tag":53,"props":324,"children":325},{},[326,328,334],{"type":50,"value":327},"This is required for gated models (e.g., ",{"type":44,"tag":91,"props":329,"children":331},{"className":330},[],[332],{"type":50,"value":333},"meta-llama\u002FMeta-Llama-3.1-8B",{"type":50,"value":335},"). For public models, the secret is optional but recommended to avoid rate limits.",{"type":44,"tag":141,"props":337,"children":339},{"id":338},"step-2-check-if-deployment-already-exists",[340],{"type":50,"value":341},"Step 2: Check if deployment already exists",{"type":44,"tag":53,"props":343,"children":344},{},[345],{"type":50,"value":346},"Before applying, check if a vLLM deployment already exists:",{"type":44,"tag":161,"props":348,"children":350},{"className":163,"code":349,"language":165,"meta":166,"style":166},"kubectl get deployment vllm -n \u003Cnamespace>\n",[351],{"type":44,"tag":91,"props":352,"children":353},{"__ignoreMap":166},[354],{"type":44,"tag":172,"props":355,"children":356},{"class":174,"line":175},[357,361,365,370,375,379,383,387,391],{"type":44,"tag":172,"props":358,"children":359},{"style":179},[360],{"type":50,"value":121},{"type":44,"tag":172,"props":362,"children":363},{"style":184},[364],{"type":50,"value":187},{"type":44,"tag":172,"props":366,"children":367},{"style":184},[368],{"type":50,"value":369}," deployment",{"type":44,"tag":172,"props":371,"children":372},{"style":184},[373],{"type":50,"value":374}," vllm",{"type":44,"tag":172,"props":376,"children":377},{"style":184},[378],{"type":50,"value":202},{"type":44,"tag":172,"props":380,"children":381},{"style":205},[382],{"type":50,"value":208},{"type":44,"tag":172,"props":384,"children":385},{"style":184},[386],{"type":50,"value":213},{"type":44,"tag":172,"props":388,"children":389},{"style":216},[390],{"type":50,"value":219},{"type":44,"tag":172,"props":392,"children":393},{"style":205},[394],{"type":50,"value":224},{"type":44,"tag":66,"props":396,"children":397},{},[398,409],{"type":44,"tag":70,"props":399,"children":400},{},[401,403,407],{"type":50,"value":402},"If it ",{"type":44,"tag":234,"props":404,"children":405},{},[406],{"type":50,"value":238},{"type":50,"value":408},": inform the user that the deployment already exists. Show the current image and status. Ask the user if they want to update it or skip.",{"type":44,"tag":70,"props":410,"children":411},{},[412,413,417],{"type":50,"value":402},{"type":44,"tag":234,"props":414,"children":415},{},[416],{"type":50,"value":249},{"type":50,"value":418},": proceed to deploy.",{"type":44,"tag":141,"props":420,"children":422},{"id":421},"step-3-deploy",[423],{"type":50,"value":424},"Step 3: Deploy",{"type":44,"tag":53,"props":426,"children":427},{},[428],{"type":50,"value":429},"Apply the template YAML files to deploy vLLM:",{"type":44,"tag":161,"props":431,"children":433},{"className":163,"code":432,"language":165,"meta":166,"style":166},"kubectl apply -f templates\u002Fvllm-service.yaml -n \u003Cnamespace>\nkubectl apply -f templates\u002Fvllm-deployment.yaml -n \u003Cnamespace>\n",[434],{"type":44,"tag":91,"props":435,"children":436},{"__ignoreMap":166},[437,479],{"type":44,"tag":172,"props":438,"children":439},{"class":174,"line":175},[440,444,449,454,459,463,467,471,475],{"type":44,"tag":172,"props":441,"children":442},{"style":179},[443],{"type":50,"value":121},{"type":44,"tag":172,"props":445,"children":446},{"style":184},[447],{"type":50,"value":448}," apply",{"type":44,"tag":172,"props":450,"children":451},{"style":184},[452],{"type":50,"value":453}," -f",{"type":44,"tag":172,"props":455,"children":456},{"style":184},[457],{"type":50,"value":458}," templates\u002Fvllm-service.yaml",{"type":44,"tag":172,"props":460,"children":461},{"style":184},[462],{"type":50,"value":202},{"type":44,"tag":172,"props":464,"children":465},{"style":205},[466],{"type":50,"value":208},{"type":44,"tag":172,"props":468,"children":469},{"style":184},[470],{"type":50,"value":213},{"type":44,"tag":172,"props":472,"children":473},{"style":216},[474],{"type":50,"value":219},{"type":44,"tag":172,"props":476,"children":477},{"style":205},[478],{"type":50,"value":224},{"type":44,"tag":172,"props":480,"children":482},{"class":174,"line":481},2,[483,487,491,495,500,504,508,512,516],{"type":44,"tag":172,"props":484,"children":485},{"style":179},[486],{"type":50,"value":121},{"type":44,"tag":172,"props":488,"children":489},{"style":184},[490],{"type":50,"value":448},{"type":44,"tag":172,"props":492,"children":493},{"style":184},[494],{"type":50,"value":453},{"type":44,"tag":172,"props":496,"children":497},{"style":184},[498],{"type":50,"value":499}," templates\u002Fvllm-deployment.yaml",{"type":44,"tag":172,"props":501,"children":502},{"style":184},[503],{"type":50,"value":202},{"type":44,"tag":172,"props":505,"children":506},{"style":205},[507],{"type":50,"value":208},{"type":44,"tag":172,"props":509,"children":510},{"style":184},[511],{"type":50,"value":213},{"type":44,"tag":172,"props":513,"children":514},{"style":216},[515],{"type":50,"value":219},{"type":44,"tag":172,"props":517,"children":518},{"style":205},[519],{"type":50,"value":224},{"type":44,"tag":141,"props":521,"children":523},{"id":522},"step-4-wait-and-verify",[524],{"type":50,"value":525},"Step 4: Wait and verify",{"type":44,"tag":53,"props":527,"children":528},{},[529],{"type":50,"value":530},"Wait for the deployment to roll out:",{"type":44,"tag":161,"props":532,"children":534},{"className":163,"code":533,"language":165,"meta":166,"style":166},"kubectl rollout status deployment\u002Fvllm -n \u003Cnamespace> --timeout=600s\n",[535],{"type":44,"tag":91,"props":536,"children":537},{"__ignoreMap":166},[538],{"type":44,"tag":172,"props":539,"children":540},{"class":174,"line":175},[541,545,550,555,560,564,568,572,576,581],{"type":44,"tag":172,"props":542,"children":543},{"style":179},[544],{"type":50,"value":121},{"type":44,"tag":172,"props":546,"children":547},{"style":184},[548],{"type":50,"value":549}," rollout",{"type":44,"tag":172,"props":551,"children":552},{"style":184},[553],{"type":50,"value":554}," status",{"type":44,"tag":172,"props":556,"children":557},{"style":184},[558],{"type":50,"value":559}," deployment\u002Fvllm",{"type":44,"tag":172,"props":561,"children":562},{"style":184},[563],{"type":50,"value":202},{"type":44,"tag":172,"props":565,"children":566},{"style":205},[567],{"type":50,"value":208},{"type":44,"tag":172,"props":569,"children":570},{"style":184},[571],{"type":50,"value":213},{"type":44,"tag":172,"props":573,"children":574},{"style":216},[575],{"type":50,"value":219},{"type":44,"tag":172,"props":577,"children":578},{"style":205},[579],{"type":50,"value":580},">",{"type":44,"tag":172,"props":582,"children":583},{"style":184},[584],{"type":50,"value":585}," --timeout=600s\n",{"type":44,"tag":53,"props":587,"children":588},{},[589],{"type":50,"value":590},"Verify the pod is running and ready:",{"type":44,"tag":161,"props":592,"children":594},{"className":163,"code":593,"language":165,"meta":166,"style":166},"kubectl get pods -n \u003Cnamespace> -l app=vllm\n",[595],{"type":44,"tag":91,"props":596,"children":597},{"__ignoreMap":166},[598],{"type":44,"tag":172,"props":599,"children":600},{"class":174,"line":175},[601,605,609,614,618,622,626,630,634,639],{"type":44,"tag":172,"props":602,"children":603},{"style":179},[604],{"type":50,"value":121},{"type":44,"tag":172,"props":606,"children":607},{"style":184},[608],{"type":50,"value":187},{"type":44,"tag":172,"props":610,"children":611},{"style":184},[612],{"type":50,"value":613}," pods",{"type":44,"tag":172,"props":615,"children":616},{"style":184},[617],{"type":50,"value":202},{"type":44,"tag":172,"props":619,"children":620},{"style":205},[621],{"type":50,"value":208},{"type":44,"tag":172,"props":623,"children":624},{"style":184},[625],{"type":50,"value":213},{"type":44,"tag":172,"props":627,"children":628},{"style":216},[629],{"type":50,"value":219},{"type":44,"tag":172,"props":631,"children":632},{"style":205},[633],{"type":50,"value":580},{"type":44,"tag":172,"props":635,"children":636},{"style":184},[637],{"type":50,"value":638}," -l",{"type":44,"tag":172,"props":640,"children":641},{"style":184},[642],{"type":50,"value":643}," app=vllm\n",{"type":44,"tag":53,"props":645,"children":646},{},[647,649,655,657,663,665,671,673,679,681,687],{"type":50,"value":648},"Confirm the pod shows ",{"type":44,"tag":91,"props":650,"children":652},{"className":651},[],[653],{"type":50,"value":654},"READY 1\u002F1",{"type":50,"value":656}," and ",{"type":44,"tag":91,"props":658,"children":660},{"className":659},[],[661],{"type":50,"value":662},"STATUS Running",{"type":50,"value":664},". If the pod is not ready yet, wait and check again. If it's in ",{"type":44,"tag":91,"props":666,"children":668},{"className":667},[],[669],{"type":50,"value":670},"CrashLoopBackOff",{"type":50,"value":672}," or ",{"type":44,"tag":91,"props":674,"children":676},{"className":675},[],[677],{"type":50,"value":678},"Error",{"type":50,"value":680},", check the logs with ",{"type":44,"tag":91,"props":682,"children":684},{"className":683},[],[685],{"type":50,"value":686},"kubectl logs -n \u003Cnamespace> -l app=vllm",{"type":50,"value":688},".",{"type":44,"tag":141,"props":690,"children":692},{"id":691},"step-5-print-deployment-summary",[693],{"type":50,"value":694},"Step 5: Print deployment summary",{"type":44,"tag":53,"props":696,"children":697},{},[698],{"type":50,"value":699},"Once the pod is ready, print a summary message to the user in this format (replace placeholders with actual values):",{"type":44,"tag":161,"props":701,"children":705},{"className":702,"code":704,"language":50},[703],"language-text","🎉 **vLLM Deployment Successful!**\n\n| Resource | Name | Status |\n|----------|------|--------|\n| Deployment | \u003Cdeployment-name> | \u003Cready>\u002F\u003Ctotal> Ready |\n| Service | \u003Cservice-name> | ClusterIP:\u003Cport> |\n| Pod | \u003Cpod-name> | Running |\n| Image | \u003Cimage> | |\n| Model | \u003Cmodel> | |\n\n \n\n**To test the API, run these two commands in your terminal:**\n\n**1. Open a port-forward** (this connects your local port \u003Cport> to the vLLM service inside the cluster):\n\nkubectl port-forward svc\u002Fvllm-svc \u003Cport>:\u003Cport> -n \u003Cnamespace>\n\n**2. In a separate terminal**, send a test request to the OpenAI-compatible API:\n\ncurl -s http:\u002F\u002Flocalhost:\u003Cport>\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"model\":\"\u003Cmodel>\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}],\"max_tokens\":50}' | python3 -m json.tool\n\nIf everything is working, you'll get a JSON response with the model's reply.\n",[706],{"type":44,"tag":91,"props":707,"children":708},{"__ignoreMap":166},[709],{"type":50,"value":704},{"type":44,"tag":59,"props":711,"children":713},{"id":712},"default-configuration",[714],{"type":50,"value":715},"Default Configuration",{"type":44,"tag":53,"props":717,"children":718},{},[719],{"type":50,"value":720},"The templates use the following defaults:",{"type":44,"tag":722,"props":723,"children":724},"table",{},[725,744],{"type":44,"tag":726,"props":727,"children":728},"thead",{},[729],{"type":44,"tag":730,"props":731,"children":732},"tr",{},[733,739],{"type":44,"tag":734,"props":735,"children":736},"th",{},[737],{"type":50,"value":738},"Parameter",{"type":44,"tag":734,"props":740,"children":741},{},[742],{"type":50,"value":743},"Default Value",{"type":44,"tag":745,"props":746,"children":747},"tbody",{},[748,765,782,799,816,832,849,865,890,914],{"type":44,"tag":730,"props":749,"children":750},{},[751,757],{"type":44,"tag":752,"props":753,"children":754},"td",{},[755],{"type":50,"value":756},"Image",{"type":44,"tag":752,"props":758,"children":759},{},[760],{"type":44,"tag":91,"props":761,"children":763},{"className":762},[],[764],{"type":50,"value":96},{"type":44,"tag":730,"props":766,"children":767},{},[768,773],{"type":44,"tag":752,"props":769,"children":770},{},[771],{"type":50,"value":772},"Model",{"type":44,"tag":752,"props":774,"children":775},{},[776],{"type":44,"tag":91,"props":777,"children":779},{"className":778},[],[780],{"type":50,"value":781},"Qwen\u002FQwen2.5-1.5B-Instruct",{"type":44,"tag":730,"props":783,"children":784},{},[785,790],{"type":44,"tag":752,"props":786,"children":787},{},[788],{"type":50,"value":789},"Port",{"type":44,"tag":752,"props":791,"children":792},{},[793],{"type":44,"tag":91,"props":794,"children":796},{"className":795},[],[797],{"type":50,"value":798},"8000",{"type":44,"tag":730,"props":800,"children":801},{},[802,807],{"type":44,"tag":752,"props":803,"children":804},{},[805],{"type":50,"value":806},"Replicas",{"type":44,"tag":752,"props":808,"children":809},{},[810],{"type":44,"tag":91,"props":811,"children":813},{"className":812},[],[814],{"type":50,"value":815},"1",{"type":44,"tag":730,"props":817,"children":818},{},[819,824],{"type":44,"tag":752,"props":820,"children":821},{},[822],{"type":50,"value":823},"GPU count",{"type":44,"tag":752,"props":825,"children":826},{},[827],{"type":44,"tag":91,"props":828,"children":830},{"className":829},[],[831],{"type":50,"value":815},{"type":44,"tag":730,"props":833,"children":834},{},[835,840],{"type":44,"tag":752,"props":836,"children":837},{},[838],{"type":50,"value":839},"GPU memory utilization",{"type":44,"tag":752,"props":841,"children":842},{},[843],{"type":44,"tag":91,"props":844,"children":846},{"className":845},[],[847],{"type":50,"value":848},"0.85",{"type":44,"tag":730,"props":850,"children":851},{},[852,857],{"type":44,"tag":752,"props":853,"children":854},{},[855],{"type":50,"value":856},"Tensor parallel size",{"type":44,"tag":752,"props":858,"children":859},{},[860],{"type":44,"tag":91,"props":861,"children":863},{"className":862},[],[864],{"type":50,"value":815},{"type":44,"tag":730,"props":866,"children":867},{},[868,873],{"type":44,"tag":752,"props":869,"children":870},{},[871],{"type":50,"value":872},"CPU request \u002F limit",{"type":44,"tag":752,"props":874,"children":875},{},[876,882,884],{"type":44,"tag":91,"props":877,"children":879},{"className":878},[],[880],{"type":50,"value":881},"12",{"type":50,"value":883}," \u002F ",{"type":44,"tag":91,"props":885,"children":887},{"className":886},[],[888],{"type":50,"value":889},"128",{"type":44,"tag":730,"props":891,"children":892},{},[893,898],{"type":44,"tag":752,"props":894,"children":895},{},[896],{"type":50,"value":897},"Memory request \u002F limit",{"type":44,"tag":752,"props":899,"children":900},{},[901,907,908],{"type":44,"tag":91,"props":902,"children":904},{"className":903},[],[905],{"type":50,"value":906},"100Gi",{"type":50,"value":883},{"type":44,"tag":91,"props":909,"children":911},{"className":910},[],[912],{"type":50,"value":913},"400Gi",{"type":44,"tag":730,"props":915,"children":916},{},[917,922],{"type":44,"tag":752,"props":918,"children":919},{},[920],{"type":50,"value":921},"Shared memory (dshm)",{"type":44,"tag":752,"props":923,"children":924},{},[925],{"type":44,"tag":91,"props":926,"children":928},{"className":927},[],[929],{"type":50,"value":930},"80Gi",{"type":44,"tag":59,"props":932,"children":934},{"id":933},"customization",[935],{"type":50,"value":936},"Customization",{"type":44,"tag":53,"props":938,"children":939},{},[940],{"type":50,"value":941},"When the user requests changes, modify the template YAML files before applying. The following can be customized:",{"type":44,"tag":66,"props":943,"children":944},{},[945,987,1011,1065,1081,1112,1128,1164,1218,1235],{"type":44,"tag":70,"props":946,"children":947},{},[948,953,955,961,963,969,971,977,979,985],{"type":44,"tag":234,"props":949,"children":950},{},[951],{"type":50,"value":952},"Image version",{"type":50,"value":954},": Change ",{"type":44,"tag":91,"props":956,"children":958},{"className":957},[],[959],{"type":50,"value":960},"image: vllm\u002Fvllm-openai:\u003Cversion>",{"type":50,"value":962}," in ",{"type":44,"tag":91,"props":964,"children":966},{"className":965},[],[967],{"type":50,"value":968},"templates\u002Fvllm-deployment.yaml",{"type":50,"value":970}," (default: ",{"type":44,"tag":91,"props":972,"children":974},{"className":973},[],[975],{"type":50,"value":976},"latest",{"type":50,"value":978},"). Use a specific version tag like ",{"type":44,"tag":91,"props":980,"children":982},{"className":981},[],[983],{"type":50,"value":984},"v0.17.1",{"type":50,"value":986}," if the user requests it.",{"type":44,"tag":70,"props":988,"children":989},{},[990,994,996,1002,1004,1010],{"type":44,"tag":234,"props":991,"children":992},{},[993],{"type":50,"value":772},{"type":50,"value":995},": Change the model name in the ",{"type":44,"tag":91,"props":997,"children":999},{"className":998},[],[1000],{"type":50,"value":1001},"vllm serve",{"type":50,"value":1003}," command inside the Deployment ",{"type":44,"tag":91,"props":1005,"children":1007},{"className":1006},[],[1008],{"type":50,"value":1009},"args",{"type":50,"value":688},{"type":44,"tag":70,"props":1012,"children":1013},{},[1014,1019,1021,1026,1028,1033,1035,1041,1043,1049,1050,1056,1057,1063],{"type":44,"tag":234,"props":1015,"children":1016},{},[1017],{"type":50,"value":1018},"Extra vLLM flags",{"type":50,"value":1020},": Append additional flags to the ",{"type":44,"tag":91,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":50,"value":1001},{"type":50,"value":1027}," command in the Deployment ",{"type":44,"tag":91,"props":1029,"children":1031},{"className":1030},[],[1032],{"type":50,"value":1009},{"type":50,"value":1034}," (e.g., ",{"type":44,"tag":91,"props":1036,"children":1038},{"className":1037},[],[1039],{"type":50,"value":1040},"--max-model-len 4096",{"type":50,"value":1042},", ",{"type":44,"tag":91,"props":1044,"children":1046},{"className":1045},[],[1047],{"type":50,"value":1048},"--kv-cache-dtype fp8",{"type":50,"value":1042},{"type":44,"tag":91,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":50,"value":1055},"--enforce-eager",{"type":50,"value":1042},{"type":44,"tag":91,"props":1058,"children":1060},{"className":1059},[],[1061],{"type":50,"value":1062},"--generation-config vllm",{"type":50,"value":1064},").",{"type":44,"tag":70,"props":1066,"children":1067},{},[1068,1072,1073,1079],{"type":44,"tag":234,"props":1069,"children":1070},{},[1071],{"type":50,"value":806},{"type":50,"value":954},{"type":44,"tag":91,"props":1074,"children":1076},{"className":1075},[],[1077],{"type":50,"value":1078},"replicas:",{"type":50,"value":1080}," in the Deployment spec.",{"type":44,"tag":70,"props":1082,"children":1083},{},[1084,1088,1089,1095,1097,1103,1104,1110],{"type":44,"tag":234,"props":1085,"children":1086},{},[1087],{"type":50,"value":823},{"type":50,"value":954},{"type":44,"tag":91,"props":1090,"children":1092},{"className":1091},[],[1093],{"type":50,"value":1094},"nvidia.com\u002Fgpu",{"type":50,"value":1096}," in both ",{"type":44,"tag":91,"props":1098,"children":1100},{"className":1099},[],[1101],{"type":50,"value":1102},"requests",{"type":50,"value":656},{"type":44,"tag":91,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":50,"value":1109},"limits",{"type":50,"value":1111}," under resources.",{"type":44,"tag":70,"props":1113,"children":1114},{},[1115,1119,1120,1126],{"type":44,"tag":234,"props":1116,"children":1117},{},[1118],{"type":50,"value":856},{"type":50,"value":954},{"type":44,"tag":91,"props":1121,"children":1123},{"className":1122},[],[1124],{"type":50,"value":1125},"--tensor-parallel-size",{"type":50,"value":1127}," flag to match the GPU count.",{"type":44,"tag":70,"props":1129,"children":1130},{},[1131,1136,1137,1143,1144,1150,1152,1157,1158,1163],{"type":44,"tag":234,"props":1132,"children":1133},{},[1134],{"type":50,"value":1135},"CPU\u002FMemory resources",{"type":50,"value":954},{"type":44,"tag":91,"props":1138,"children":1140},{"className":1139},[],[1141],{"type":50,"value":1142},"cpu",{"type":50,"value":656},{"type":44,"tag":91,"props":1145,"children":1147},{"className":1146},[],[1148],{"type":50,"value":1149},"memory",{"type":50,"value":1151}," values under ",{"type":44,"tag":91,"props":1153,"children":1155},{"className":1154},[],[1156],{"type":50,"value":1102},{"type":50,"value":656},{"type":44,"tag":91,"props":1159,"children":1161},{"className":1160},[],[1162],{"type":50,"value":1109},{"type":50,"value":688},{"type":44,"tag":70,"props":1165,"children":1166},{},[1167,1171,1172,1178,1180,1186,1188,1194,1196,1201,1203,1209,1211,1216],{"type":44,"tag":234,"props":1168,"children":1169},{},[1170],{"type":50,"value":789},{"type":50,"value":954},{"type":44,"tag":91,"props":1173,"children":1175},{"className":1174},[],[1176],{"type":50,"value":1177},"containerPort",{"type":50,"value":1179}," in the Deployment, ",{"type":44,"tag":91,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":50,"value":1185},"port",{"type":50,"value":1187},"\u002F",{"type":44,"tag":91,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":50,"value":1193},"targetPort",{"type":50,"value":1195}," in the Service, the ",{"type":44,"tag":91,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":50,"value":1185},{"type":50,"value":1202}," in all health probes (liveness, readiness, startup), AND add ",{"type":44,"tag":91,"props":1204,"children":1206},{"className":1205},[],[1207],{"type":50,"value":1208},"--port \u003Cport>",{"type":50,"value":1210}," to the ",{"type":44,"tag":91,"props":1212,"children":1214},{"className":1213},[],[1215],{"type":50,"value":1001},{"type":50,"value":1217}," command in args. All four must match.",{"type":44,"tag":70,"props":1219,"children":1220},{},[1221,1226,1228,1234],{"type":44,"tag":234,"props":1222,"children":1223},{},[1224],{"type":50,"value":1225},"Namespace",{"type":50,"value":1227},": Apply to a specific namespace using ",{"type":44,"tag":91,"props":1229,"children":1231},{"className":1230},[],[1232],{"type":50,"value":1233},"-n \u003Cnamespace>",{"type":50,"value":688},{"type":44,"tag":70,"props":1236,"children":1237},{},[1238,1243,1245,1251,1253,1259],{"type":44,"tag":234,"props":1239,"children":1240},{},[1241],{"type":50,"value":1242},"Shared memory size",{"type":50,"value":1244},": Change the ",{"type":44,"tag":91,"props":1246,"children":1248},{"className":1247},[],[1249],{"type":50,"value":1250},"sizeLimit",{"type":50,"value":1252}," of the ",{"type":44,"tag":91,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":50,"value":1258},"dshm",{"type":50,"value":1260}," emptyDir volume.",{"type":44,"tag":53,"props":1262,"children":1263},{},[1264],{"type":50,"value":1265},"Edit the template files using the Edit tool, then apply the modified templates.",{"type":44,"tag":59,"props":1267,"children":1269},{"id":1268},"status-check",[1270],{"type":50,"value":1271},"Status Check",{"type":44,"tag":161,"props":1273,"children":1275},{"className":163,"code":1274,"language":165,"meta":166,"style":166},"kubectl get deployment,svc,pods -n \u003Cnamespace> -l app=vllm\n",[1276],{"type":44,"tag":91,"props":1277,"children":1278},{"__ignoreMap":166},[1279],{"type":44,"tag":172,"props":1280,"children":1281},{"class":174,"line":175},[1282,1286,1290,1295,1299,1303,1307,1311,1315,1319],{"type":44,"tag":172,"props":1283,"children":1284},{"style":179},[1285],{"type":50,"value":121},{"type":44,"tag":172,"props":1287,"children":1288},{"style":184},[1289],{"type":50,"value":187},{"type":44,"tag":172,"props":1291,"children":1292},{"style":184},[1293],{"type":50,"value":1294}," deployment,svc,pods",{"type":44,"tag":172,"props":1296,"children":1297},{"style":184},[1298],{"type":50,"value":202},{"type":44,"tag":172,"props":1300,"children":1301},{"style":205},[1302],{"type":50,"value":208},{"type":44,"tag":172,"props":1304,"children":1305},{"style":184},[1306],{"type":50,"value":213},{"type":44,"tag":172,"props":1308,"children":1309},{"style":216},[1310],{"type":50,"value":219},{"type":44,"tag":172,"props":1312,"children":1313},{"style":205},[1314],{"type":50,"value":580},{"type":44,"tag":172,"props":1316,"children":1317},{"style":184},[1318],{"type":50,"value":638},{"type":44,"tag":172,"props":1320,"children":1321},{"style":184},[1322],{"type":50,"value":643},{"type":44,"tag":59,"props":1324,"children":1326},{"id":1325},"cleanup",[1327],{"type":50,"value":1328},"Cleanup",{"type":44,"tag":53,"props":1330,"children":1331},{},[1332],{"type":50,"value":1333},"When the user asks to clean up or delete the vLLM deployment, run the following steps:",{"type":44,"tag":1335,"props":1336,"children":1337},"ol",{},[1338],{"type":44,"tag":70,"props":1339,"children":1340},{},[1341],{"type":50,"value":1342},"Delete the Deployment and Service:",{"type":44,"tag":161,"props":1344,"children":1346},{"className":163,"code":1345,"language":165,"meta":166,"style":166},"kubectl delete -f templates\u002Fvllm-deployment.yaml -n \u003Cnamespace>\nkubectl delete -f templates\u002Fvllm-service.yaml -n \u003Cnamespace>\n",[1347],{"type":44,"tag":91,"props":1348,"children":1349},{"__ignoreMap":166},[1350,1390],{"type":44,"tag":172,"props":1351,"children":1352},{"class":174,"line":175},[1353,1357,1362,1366,1370,1374,1378,1382,1386],{"type":44,"tag":172,"props":1354,"children":1355},{"style":179},[1356],{"type":50,"value":121},{"type":44,"tag":172,"props":1358,"children":1359},{"style":184},[1360],{"type":50,"value":1361}," delete",{"type":44,"tag":172,"props":1363,"children":1364},{"style":184},[1365],{"type":50,"value":453},{"type":44,"tag":172,"props":1367,"children":1368},{"style":184},[1369],{"type":50,"value":499},{"type":44,"tag":172,"props":1371,"children":1372},{"style":184},[1373],{"type":50,"value":202},{"type":44,"tag":172,"props":1375,"children":1376},{"style":205},[1377],{"type":50,"value":208},{"type":44,"tag":172,"props":1379,"children":1380},{"style":184},[1381],{"type":50,"value":213},{"type":44,"tag":172,"props":1383,"children":1384},{"style":216},[1385],{"type":50,"value":219},{"type":44,"tag":172,"props":1387,"children":1388},{"style":205},[1389],{"type":50,"value":224},{"type":44,"tag":172,"props":1391,"children":1392},{"class":174,"line":481},[1393,1397,1401,1405,1409,1413,1417,1421,1425],{"type":44,"tag":172,"props":1394,"children":1395},{"style":179},[1396],{"type":50,"value":121},{"type":44,"tag":172,"props":1398,"children":1399},{"style":184},[1400],{"type":50,"value":1361},{"type":44,"tag":172,"props":1402,"children":1403},{"style":184},[1404],{"type":50,"value":453},{"type":44,"tag":172,"props":1406,"children":1407},{"style":184},[1408],{"type":50,"value":458},{"type":44,"tag":172,"props":1410,"children":1411},{"style":184},[1412],{"type":50,"value":202},{"type":44,"tag":172,"props":1414,"children":1415},{"style":205},[1416],{"type":50,"value":208},{"type":44,"tag":172,"props":1418,"children":1419},{"style":184},[1420],{"type":50,"value":213},{"type":44,"tag":172,"props":1422,"children":1423},{"style":216},[1424],{"type":50,"value":219},{"type":44,"tag":172,"props":1426,"children":1427},{"style":205},[1428],{"type":50,"value":224},{"type":44,"tag":1335,"props":1430,"children":1431},{"start":481},[1432],{"type":44,"tag":70,"props":1433,"children":1434},{},[1435],{"type":50,"value":1436},"Ask the user if they also want to delete the HF token secret. If yes:",{"type":44,"tag":161,"props":1438,"children":1440},{"className":163,"code":1439,"language":165,"meta":166,"style":166},"kubectl delete secret hf-token -n \u003Cnamespace>\n",[1441],{"type":44,"tag":91,"props":1442,"children":1443},{"__ignoreMap":166},[1444],{"type":44,"tag":172,"props":1445,"children":1446},{"class":174,"line":175},[1447,1451,1455,1459,1463,1467,1471,1475,1479],{"type":44,"tag":172,"props":1448,"children":1449},{"style":179},[1450],{"type":50,"value":121},{"type":44,"tag":172,"props":1452,"children":1453},{"style":184},[1454],{"type":50,"value":1361},{"type":44,"tag":172,"props":1456,"children":1457},{"style":184},[1458],{"type":50,"value":192},{"type":44,"tag":172,"props":1460,"children":1461},{"style":184},[1462],{"type":50,"value":197},{"type":44,"tag":172,"props":1464,"children":1465},{"style":184},[1466],{"type":50,"value":202},{"type":44,"tag":172,"props":1468,"children":1469},{"style":205},[1470],{"type":50,"value":208},{"type":44,"tag":172,"props":1472,"children":1473},{"style":184},[1474],{"type":50,"value":213},{"type":44,"tag":172,"props":1476,"children":1477},{"style":216},[1478],{"type":50,"value":219},{"type":44,"tag":172,"props":1480,"children":1481},{"style":205},[1482],{"type":50,"value":224},{"type":44,"tag":1335,"props":1484,"children":1486},{"start":1485},3,[1487],{"type":44,"tag":70,"props":1488,"children":1489},{},[1490],{"type":50,"value":1491},"Verify everything is cleaned up:",{"type":44,"tag":161,"props":1493,"children":1494},{"className":163,"code":1274,"language":165,"meta":166,"style":166},[1495],{"type":44,"tag":91,"props":1496,"children":1497},{"__ignoreMap":166},[1498],{"type":44,"tag":172,"props":1499,"children":1500},{"class":174,"line":175},[1501,1505,1509,1513,1517,1521,1525,1529,1533,1537],{"type":44,"tag":172,"props":1502,"children":1503},{"style":179},[1504],{"type":50,"value":121},{"type":44,"tag":172,"props":1506,"children":1507},{"style":184},[1508],{"type":50,"value":187},{"type":44,"tag":172,"props":1510,"children":1511},{"style":184},[1512],{"type":50,"value":1294},{"type":44,"tag":172,"props":1514,"children":1515},{"style":184},[1516],{"type":50,"value":202},{"type":44,"tag":172,"props":1518,"children":1519},{"style":205},[1520],{"type":50,"value":208},{"type":44,"tag":172,"props":1522,"children":1523},{"style":184},[1524],{"type":50,"value":213},{"type":44,"tag":172,"props":1526,"children":1527},{"style":216},[1528],{"type":50,"value":219},{"type":44,"tag":172,"props":1530,"children":1531},{"style":205},[1532],{"type":50,"value":580},{"type":44,"tag":172,"props":1534,"children":1535},{"style":184},[1536],{"type":50,"value":638},{"type":44,"tag":172,"props":1538,"children":1539},{"style":184},[1540],{"type":50,"value":643},{"type":44,"tag":1335,"props":1542,"children":1544},{"start":1543},4,[1545],{"type":44,"tag":70,"props":1546,"children":1547},{},[1548],{"type":50,"value":1549},"Print a summary message to the user:",{"type":44,"tag":161,"props":1551,"children":1554},{"className":1552,"code":1553,"language":50},[703],"vLLM deployment has been cleaned up from namespace \u003Cnamespace>.\nDeleted: Deployment\u002Fvllm, Service\u002Fvllm-svc\nHF token secret: \u003Ckept\u002Fdeleted>\n",[1555],{"type":44,"tag":91,"props":1556,"children":1557},{"__ignoreMap":166},[1558],{"type":50,"value":1553},{"type":44,"tag":59,"props":1560,"children":1562},{"id":1561},"troubleshooting",[1563],{"type":50,"value":1564},"Troubleshooting",{"type":44,"tag":66,"props":1566,"children":1567},{},[1568,1586,1603,1613,1646,1664],{"type":44,"tag":70,"props":1569,"children":1570},{},[1571,1576,1578,1584],{"type":44,"tag":234,"props":1572,"children":1573},{},[1574],{"type":50,"value":1575},"Pod stuck in Pending",{"type":50,"value":1577},": No GPU nodes available. Check ",{"type":44,"tag":91,"props":1579,"children":1581},{"className":1580},[],[1582],{"type":50,"value":1583},"kubectl describe pod \u003Cpod-name>",{"type":50,"value":1585}," for scheduling errors. Ensure NVIDIA GPU Operator or device plugin is installed.",{"type":44,"tag":70,"props":1587,"children":1588},{},[1589,1594,1596,1601],{"type":44,"tag":234,"props":1590,"children":1591},{},[1592],{"type":50,"value":1593},"Pod OOMKilled",{"type":50,"value":1595},": Increase ",{"type":44,"tag":91,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":50,"value":1149},{"type":50,"value":1602}," limits in the Deployment, or use a smaller model.",{"type":44,"tag":70,"props":1604,"children":1605},{},[1606,1611],{"type":44,"tag":234,"props":1607,"children":1608},{},[1609],{"type":50,"value":1610},"ImagePullBackOff",{"type":50,"value":1612},": Check the image name and tag. Verify the node has access to Docker Hub \u002F the container registry.",{"type":44,"tag":70,"props":1614,"children":1615},{},[1616,1621,1623,1629,1631,1636,1638,1644],{"type":44,"tag":234,"props":1617,"children":1618},{},[1619],{"type":50,"value":1620},"Startup probe failures (CrashLoopBackOff)",{"type":50,"value":1622},": Model download may be slow. Check logs with ",{"type":44,"tag":91,"props":1624,"children":1626},{"className":1625},[],[1627],{"type":50,"value":1628},"kubectl logs \u003Cpod-name>",{"type":50,"value":1630},". Ensure ",{"type":44,"tag":91,"props":1632,"children":1634},{"className":1633},[],[1635],{"type":50,"value":157},{"type":50,"value":1637}," secret exists for gated models. Increase ",{"type":44,"tag":91,"props":1639,"children":1641},{"className":1640},[],[1642],{"type":50,"value":1643},"failureThreshold",{"type":50,"value":1645}," on the startup probe if needed.",{"type":44,"tag":70,"props":1647,"children":1648},{},[1649,1654,1656,1662],{"type":44,"tag":234,"props":1650,"children":1651},{},[1652],{"type":50,"value":1653},"HF_TOKEN not working",{"type":50,"value":1655},": Verify the secret exists: ",{"type":44,"tag":91,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":50,"value":1661},"kubectl get secret hf-token -n \u003Cnamespace>",{"type":50,"value":1663},". Check the token is valid.",{"type":44,"tag":70,"props":1665,"children":1666},{},[1667,1672,1674,1679],{"type":44,"tag":234,"props":1668,"children":1669},{},[1670],{"type":50,"value":1671},"GPU not detected in container",{"type":50,"value":1673},": Ensure ",{"type":44,"tag":91,"props":1675,"children":1677},{"className":1676},[],[1678],{"type":50,"value":1094},{"type":50,"value":1680}," resource is requested and the NVIDIA device plugin is running on the node.",{"type":44,"tag":59,"props":1682,"children":1684},{"id":1683},"references",[1685],{"type":50,"value":1686},"References",{"type":44,"tag":66,"props":1688,"children":1689},{},[1690,1702,1712,1722],{"type":44,"tag":70,"props":1691,"children":1692},{},[1693],{"type":44,"tag":1694,"props":1695,"children":1699},"a",{"href":1696,"rel":1697},"https:\u002F\u002Fdocs.vllm.ai\u002F",[1698],"nofollow",[1700],{"type":50,"value":1701},"vLLM Documentation",{"type":44,"tag":70,"props":1703,"children":1704},{},[1705],{"type":44,"tag":1694,"props":1706,"children":1709},{"href":1707,"rel":1708},"https:\u002F\u002Fhub.docker.com\u002Fr\u002Fvllm\u002Fvllm-openai\u002Ftags",[1698],[1710],{"type":50,"value":1711},"vLLM Docker Images",{"type":44,"tag":70,"props":1713,"children":1714},{},[1715],{"type":44,"tag":1694,"props":1716,"children":1719},{"href":1717,"rel":1718},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Findex.html",[1698],[1720],{"type":50,"value":1721},"NVIDIA GPU Operator",{"type":44,"tag":70,"props":1723,"children":1724},{},[1725],{"type":44,"tag":1694,"props":1726,"children":1729},{"href":1727,"rel":1728},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fmanage-gpus\u002Fscheduling-gpus\u002F",[1698],[1730],{"type":50,"value":1731},"Kubernetes GPU Scheduling",{"type":44,"tag":1733,"props":1734,"children":1735},"style",{},[1736],{"type":50,"value":1737},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1739,"total":1829},[1740,1757,1772,1786,1799,1807,1818],{"slug":1741,"name":1741,"fn":1742,"description":1743,"org":1744,"tags":1745,"stars":1754,"repoUrl":1755,"updatedAt":1756},"openclaw-vsr-bridge","integrate vLLM Semantic Router","Install vLLM Semantic Router in agent-safe mode, import supported OpenClaw model providers into canonical VSR config, and rewrite OpenClaw to target VSR.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1746,1749,1750,1753],{"name":1747,"slug":1748,"type":16},"Agents","agents",{"name":14,"slug":15,"type":16},{"name":1751,"slug":1752,"type":16},"Routing","routing",{"name":9,"slug":8,"type":16},4962,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fsemantic-router","2026-07-18T05:48:09.091161",{"slug":1758,"name":1758,"fn":1759,"description":1760,"org":1761,"tags":1762,"stars":27,"repoUrl":28,"updatedAt":1771},"vllm-bench-random-synthetic","run vLLM synthetic performance benchmarks","Run vLLM performance benchmark using synthetic random data to measure throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and other key performance metrics. Use when the user wants to quickly test vLLM serving performance without downloading external datasets.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1763,1766,1767,1770],{"name":1764,"slug":1765,"type":16},"Benchmarking","benchmarking",{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},"Performance","performance",{"name":9,"slug":8,"type":16},"2026-07-18T05:46:59.036927",{"slug":1773,"name":1773,"fn":1774,"description":1775,"org":1776,"tags":1777,"stars":27,"repoUrl":28,"updatedAt":1785},"vllm-bench-serve","benchmark LLM serving endpoints","Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve. Supports multiple datasets (random, sharegpt, sonnet, HF), backends (openai, openai-chat, vllm-pooling, embeddings), throughput\u002Flatency testing with request-rate control, and result saving. Use when benchmarking LLM serving performance, measuring TTFT\u002FTPOT, or load testing inference APIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1778,1781,1782,1783,1784],{"name":1779,"slug":1780,"type":16},"API Development","api-development",{"name":1764,"slug":1765,"type":16},{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:02.283164",{"slug":1787,"name":1787,"fn":1788,"description":1789,"org":1790,"tags":1791,"stars":27,"repoUrl":28,"updatedAt":1798},"vllm-deploy-docker","deploy vLLM using Docker containers","Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1792,1793,1796,1797],{"name":19,"slug":20,"type":16},{"name":1794,"slug":1795,"type":16},"Docker","docker",{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:01.814419",{"slug":4,"name":4,"fn":5,"description":6,"org":1800,"tags":1801,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1802,1803,1804,1805,1806],{"name":22,"slug":23,"type":16},{"name":19,"slug":20,"type":16},{"name":25,"slug":26,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":1808,"name":1808,"fn":1809,"description":1810,"org":1811,"tags":1812,"stars":27,"repoUrl":28,"updatedAt":1817},"vllm-deploy-simple","deploy and test vLLM servers","Quick install and deploy vLLM, start serving with a simple LLM, and test OpenAI API.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1813,1814,1815,1816],{"name":1779,"slug":1780,"type":16},{"name":19,"slug":20,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:01.359655",{"slug":1819,"name":1819,"fn":1820,"description":1821,"org":1822,"tags":1823,"stars":27,"repoUrl":28,"updatedAt":1828},"vllm-prefix-cache-bench","benchmark vLLM prefix caching efficiency","This is a skill for benchmarking the efficiency of automatic prefix caching in vLLM using fixed prompts, real-world datasets, or synthetic prefix\u002Fsuffix patterns. Use when the user asks to benchmark prefix caching hit rate, caching efficiency, or repeated-prompt performance in vLLM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1824,1825,1826,1827],{"name":1764,"slug":1765,"type":16},{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:48:07.243092",7,{"items":1831,"total":1876},[1832,1839,1847,1854,1862,1869],{"slug":1758,"name":1758,"fn":1759,"description":1760,"org":1833,"tags":1834,"stars":27,"repoUrl":28,"updatedAt":1771},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1835,1836,1837,1838],{"name":1764,"slug":1765,"type":16},{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},{"name":9,"slug":8,"type":16},{"slug":1773,"name":1773,"fn":1774,"description":1775,"org":1840,"tags":1841,"stars":27,"repoUrl":28,"updatedAt":1785},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1842,1843,1844,1845,1846],{"name":1779,"slug":1780,"type":16},{"name":1764,"slug":1765,"type":16},{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},{"name":9,"slug":8,"type":16},{"slug":1787,"name":1787,"fn":1788,"description":1789,"org":1848,"tags":1849,"stars":27,"repoUrl":28,"updatedAt":1798},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1850,1851,1852,1853],{"name":19,"slug":20,"type":16},{"name":1794,"slug":1795,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":1855,"tags":1856,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1857,1858,1859,1860,1861],{"name":22,"slug":23,"type":16},{"name":19,"slug":20,"type":16},{"name":25,"slug":26,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":1808,"name":1808,"fn":1809,"description":1810,"org":1863,"tags":1864,"stars":27,"repoUrl":28,"updatedAt":1817},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1865,1866,1867,1868],{"name":1779,"slug":1780,"type":16},{"name":19,"slug":20,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":1819,"name":1819,"fn":1820,"description":1821,"org":1870,"tags":1871,"stars":27,"repoUrl":28,"updatedAt":1828},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1872,1873,1874,1875],{"name":1764,"slug":1765,"type":16},{"name":14,"slug":15,"type":16},{"name":1768,"slug":1769,"type":16},{"name":9,"slug":8,"type":16},6]