[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-vllm-vllm-bench-serve":3,"mdc-3ooqen-key":38,"related-org-vllm-vllm-bench-serve":2409,"related-repo-vllm-vllm-bench-serve":2499},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"vllm-bench-serve","benchmark LLM serving endpoints","Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve. Supports multiple datasets (random, sharegpt, sonnet, HF), backends (openai, openai-chat, vllm-pooling, embeddings), throughput\u002Flatency testing with request-rate control, and result saving. Use when benchmarking LLM serving performance, measuring TTFT\u002FTPOT, or load testing inference APIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"vllm","vLLM","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fvllm.png","vllm-project",[13,17,20,23,24],{"name":14,"slug":15,"type":16},"Performance","performance","tag",{"name":18,"slug":19,"type":16},"Benchmarking","benchmarking",{"name":21,"slug":22,"type":16},"LLM","llm",{"name":9,"slug":8,"type":16},{"name":25,"slug":26,"type":16},"API Development","api-development",88,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills","2026-07-18T05:47:02.283164",null,23,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Agent skills for vLLM","https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fvllm-skills\u002Fskills\u002Fvllm-bench-serve","---\nname: vllm-bench-serve\ndescription: Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve. Supports multiple datasets (random, sharegpt, sonnet, HF), backends (openai, openai-chat, vllm-pooling, embeddings), throughput\u002Flatency testing with request-rate control, and result saving. Use when benchmarking LLM serving performance, measuring TTFT\u002FTPOT, or load testing inference APIs.\n---\n\n# vLLM Bench Serve\n\nBenchmark vLLM or any OpenAI-compatible serving endpoint using the `vllm bench serve` CLI. Measures throughput, latency (TTFT, TPOT), and goodput against configurable request load.\n\nReference: [vLLM Bench Serve Documentation](https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fbench\u002Fserve\u002F)\n\n## Prerequisites\n\n- vLLM installed (or any OpenAI-compatible server running)\n- A vLLM server or API endpoint already serving a model\n- Python environment with vLLM for the benchmark client\n\n## Quick Start\n\n**Basic benchmark against local vLLM server (default random dataset, 1000 prompts):**\n\n```bash\nvllm bench serve \\\n  --backend openai-chat \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions\n```\n\n**Save results to JSON:**\n\n```bash\nvllm bench serve \\\n  --backend openai-chat \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --save-result \\\n  --result-dir .\u002Fbench-results \\\n  --metadata \"version=0.6.0\" \"tp=1\"\n```\n\n> **Note:** When using `--backend openai-chat`, you must specify `--endpoint \u002Fv1\u002Fchat\u002Fcompletions` (default is `\u002Fv1\u002Fcompletions`).\n\n## Core Arguments\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--backend` | `openai` | Backend type: `openai`, `openai-chat`, `openai-embeddings`, `vllm`, `vllm-pooling`, `vllm-rerank`, etc. |\n| `--host` | `127.0.0.1` | Server host |\n| `--port` | `8000` | Server port |\n| `--base-url` | - | Alternative: full base URL instead of host:port |\n| `--endpoint` | `\u002Fv1\u002Fcompletions` | API endpoint; use `\u002Fv1\u002Fchat\u002Fcompletions` for openai-chat |\n| `--model` | (from \u002Fv1\u002Fmodels) | Model name |\n| `--num-prompts` | `1000` | Number of prompts to process |\n| `--request-rate` | `inf` | Requests per second; `inf` = burst all at once |\n| `--max-concurrency` | - | Max concurrent requests (caps parallelism) |\n| `--num-warmups` | `0` | Warmup requests before measuring |\n\n## Datasets\n\n| `--dataset-name` | Use Case |\n|------------------|----------|\n| `random` | Synthetic random prompts (default) |\n| `sharegpt` | ShareGPT conversation format; requires `--dataset-path` |\n| `sonnet` | Sonnet-style prompts |\n| `hf` | HuggingFace dataset; requires `--dataset-path` (dataset ID) |\n| `custom` \u002F `custom_mm` | Custom dataset; requires `--dataset-path` |\n| `prefix_repetition` | Prefix repetition benchmark |\n| `random-mm` | Random multimodal (images\u002Fvideos) |\n| `spec_bench` | Spec bench dataset |\n\n**Dataset-specific options (examples):**\n\n```bash\n# Random: control input\u002Foutput length\n--dataset-name random --random-input-len 1024 --random-output-len 128\n\n# Sonnet defaults: input 550, output 150, prefix 200\n--dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150\n\n# HuggingFace dataset\n--dataset-name hf --dataset-path \"lmarena-ai\u002FVisionArena-Chat\" --hf-split test\n\n# General overrides (map to dataset-specific args)\n--input-len 512 --output-len 256\n```\n\n## Load Control\n\n```bash\n# Fixed request rate (Poisson process)\n--request-rate 10\n\n# More bursty arrivals (gamma distribution, burstiness \u003C 1)\n--request-rate 10 --burstiness 0.5\n\n# Ramp-up from low to high RPS\n--ramp-up-strategy linear --ramp-up-start-rps 1 --ramp-up-end-rps 50\n\n# Limit concurrency (useful for rate-limited APIs)\n--max-concurrency 32\n```\n\n## Results and Metrics\n\n| Argument | Description |\n|----------|-------------|\n| `--save-result` | Save benchmark results to JSON |\n| `--save-detailed` | Include per-request TTFT, TPOT, errors in JSON |\n| `--append-result` | Append to existing result file |\n| `--result-dir` | Directory for result files |\n| `--result-filename` | Custom filename (default: `{label}-{request_rate}qps-{model}-{timestamp}.json`) |\n| `--percentile-metrics` | Metrics for percentiles: `ttft`, `tpot`, `itl`, `e2el` (default: `ttft,tpot,itl`) |\n| `--metric-percentiles` | Percentile values, e.g. `25,50,99` (default: `99`) |\n| `--goodput` | SLO for goodput: `ttft:500 tpot:50` (ms) |\n\n## Sampling Parameters (OpenAI-compatible backends)\n\n```bash\n--temperature 0.7 --top-p 0.95 --top-k 50\n--frequency-penalty 0 --presence-penalty 0 --repetition-penalty 1.0\n```\n\n## Common Workflows\n\n**1. Throughput test with random dataset (burst):**\n\n```bash\nvllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --dataset-name random \\\n  --num-prompts 500 --random-input-len 512 --random-output-len 128\n```\n\n**2. Latency test with fixed QPS:**\n\n```bash\nvllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --request-rate 5 --num-prompts 200 \\\n  --save-result --percentile-metrics ttft,tpot --metric-percentiles 50,99\n```\n\n**3. Benchmark against remote API (base-url):**\n\n```bash\nvllm bench serve --backend openai-chat \\\n  --base-url \"https:\u002F\u002Fapi.example.com\u002Fv1\" \\\n  --model my-model \\\n  --header \"Authorization=Bearer $API_KEY\"\n```\n\n**4. Run inside Docker (when vLLM client not on host):**\n\n```bash\ndocker exec \u003Ccontainer-name> vllm bench serve \\\n  --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --dataset-name random --num-prompts 100\n```\n\n## Troubleshooting\n\n- **Connection refused**: Ensure the server is running and `--host`\u002F`--port` or `--base-url` are correct.\n- **Model not found**: Pass `--model` explicitly or ensure `\u002Fv1\u002Fmodels` returns the model.\n- **URL must end with chat\u002Fcompletions**: Use `--endpoint \u002Fv1\u002Fchat\u002Fcompletions` when `--backend openai-chat`.\n- **Rate limit \u002F 429**: Reduce `--request-rate` or `--max-concurrency`.\n- **Ready check**: Use `--ready-check-timeout-sec 60` to wait for the endpoint before benchmarking.\n- **SSL**: Use `--insecure` for self-signed certificates.\n\n## Notes\n\n- For embeddings\u002Frerank benchmarks, use `--backend openai-embeddings`, `vllm-pooling`, or `vllm-rerank`.\n- `--profile` requires `--profiler-config` on the server for vLLM profiling.\n- Goodput SLOs are useful for SLA-style analysis; see [DistServe paper](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2401.09670) for details.",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,51,66,80,87,107,113,122,248,256,428,466,472,808,814,1002,1010,1202,1208,1347,1353,1575,1581,1652,1658,1666,1794,1802,1936,1944,2045,2053,2191,2197,2336,2342,2403],{"type":44,"tag":45,"props":46,"children":47},"element","h1",{"id":4},[48],{"type":49,"value":50},"text","vLLM Bench Serve",{"type":44,"tag":52,"props":53,"children":54},"p",{},[55,57,64],{"type":49,"value":56},"Benchmark vLLM or any OpenAI-compatible serving endpoint using the ",{"type":44,"tag":58,"props":59,"children":61},"code",{"className":60},[],[62],{"type":49,"value":63},"vllm bench serve",{"type":49,"value":65}," CLI. Measures throughput, latency (TTFT, TPOT), and goodput against configurable request load.",{"type":44,"tag":52,"props":67,"children":68},{},[69,71],{"type":49,"value":70},"Reference: ",{"type":44,"tag":72,"props":73,"children":77},"a",{"href":74,"rel":75},"https:\u002F\u002Fdocs.vllm.ai\u002Fen\u002Flatest\u002Fcli\u002Fbench\u002Fserve\u002F",[76],"nofollow",[78],{"type":49,"value":79},"vLLM Bench Serve Documentation",{"type":44,"tag":81,"props":82,"children":84},"h2",{"id":83},"prerequisites",[85],{"type":49,"value":86},"Prerequisites",{"type":44,"tag":88,"props":89,"children":90},"ul",{},[91,97,102],{"type":44,"tag":92,"props":93,"children":94},"li",{},[95],{"type":49,"value":96},"vLLM installed (or any OpenAI-compatible server running)",{"type":44,"tag":92,"props":98,"children":99},{},[100],{"type":49,"value":101},"A vLLM server or API endpoint already serving a model",{"type":44,"tag":92,"props":103,"children":104},{},[105],{"type":49,"value":106},"Python environment with vLLM for the benchmark client",{"type":44,"tag":81,"props":108,"children":110},{"id":109},"quick-start",[111],{"type":49,"value":112},"Quick Start",{"type":44,"tag":52,"props":114,"children":115},{},[116],{"type":44,"tag":117,"props":118,"children":119},"strong",{},[120],{"type":49,"value":121},"Basic benchmark against local vLLM server (default random dataset, 1000 prompts):",{"type":44,"tag":123,"props":124,"children":129},"pre",{"className":125,"code":126,"language":127,"meta":128,"style":128},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","vllm bench serve \\\n  --backend openai-chat \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions\n","bash","",[130],{"type":44,"tag":58,"props":131,"children":132},{"__ignoreMap":128},[133,161,179,198,216,234],{"type":44,"tag":134,"props":135,"children":138},"span",{"class":136,"line":137},"line",1,[139,144,150,155],{"type":44,"tag":134,"props":140,"children":142},{"style":141},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[143],{"type":49,"value":8},{"type":44,"tag":134,"props":145,"children":147},{"style":146},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[148],{"type":49,"value":149}," bench",{"type":44,"tag":134,"props":151,"children":152},{"style":146},[153],{"type":49,"value":154}," serve",{"type":44,"tag":134,"props":156,"children":158},{"style":157},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[159],{"type":49,"value":160}," \\\n",{"type":44,"tag":134,"props":162,"children":164},{"class":136,"line":163},2,[165,170,175],{"type":44,"tag":134,"props":166,"children":167},{"style":146},[168],{"type":49,"value":169},"  --backend",{"type":44,"tag":134,"props":171,"children":172},{"style":146},[173],{"type":49,"value":174}," openai-chat",{"type":44,"tag":134,"props":176,"children":177},{"style":157},[178],{"type":49,"value":160},{"type":44,"tag":134,"props":180,"children":182},{"class":136,"line":181},3,[183,188,194],{"type":44,"tag":134,"props":184,"children":185},{"style":146},[186],{"type":49,"value":187},"  --host",{"type":44,"tag":134,"props":189,"children":191},{"style":190},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[192],{"type":49,"value":193}," 127.0.0.1",{"type":44,"tag":134,"props":195,"children":196},{"style":157},[197],{"type":49,"value":160},{"type":44,"tag":134,"props":199,"children":201},{"class":136,"line":200},4,[202,207,212],{"type":44,"tag":134,"props":203,"children":204},{"style":146},[205],{"type":49,"value":206},"  --port",{"type":44,"tag":134,"props":208,"children":209},{"style":190},[210],{"type":49,"value":211}," 8000",{"type":44,"tag":134,"props":213,"children":214},{"style":157},[215],{"type":49,"value":160},{"type":44,"tag":134,"props":217,"children":219},{"class":136,"line":218},5,[220,225,230],{"type":44,"tag":134,"props":221,"children":222},{"style":146},[223],{"type":49,"value":224},"  --model",{"type":44,"tag":134,"props":226,"children":227},{"style":146},[228],{"type":49,"value":229}," Qwen\u002FQwen2.5-1.5B-Instruct",{"type":44,"tag":134,"props":231,"children":232},{"style":157},[233],{"type":49,"value":160},{"type":44,"tag":134,"props":235,"children":237},{"class":136,"line":236},6,[238,243],{"type":44,"tag":134,"props":239,"children":240},{"style":146},[241],{"type":49,"value":242},"  --endpoint",{"type":44,"tag":134,"props":244,"children":245},{"style":146},[246],{"type":49,"value":247}," \u002Fv1\u002Fchat\u002Fcompletions\n",{"type":44,"tag":52,"props":249,"children":250},{},[251],{"type":44,"tag":117,"props":252,"children":253},{},[254],{"type":49,"value":255},"Save results to JSON:",{"type":44,"tag":123,"props":257,"children":259},{"className":125,"code":258,"language":127,"meta":128,"style":128},"vllm bench serve \\\n  --backend openai-chat \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --save-result \\\n  --result-dir .\u002Fbench-results \\\n  --metadata \"version=0.6.0\" \"tp=1\"\n",[260],{"type":44,"tag":58,"props":261,"children":262},{"__ignoreMap":128},[263,282,297,312,327,342,358,371,389],{"type":44,"tag":134,"props":264,"children":265},{"class":136,"line":137},[266,270,274,278],{"type":44,"tag":134,"props":267,"children":268},{"style":141},[269],{"type":49,"value":8},{"type":44,"tag":134,"props":271,"children":272},{"style":146},[273],{"type":49,"value":149},{"type":44,"tag":134,"props":275,"children":276},{"style":146},[277],{"type":49,"value":154},{"type":44,"tag":134,"props":279,"children":280},{"style":157},[281],{"type":49,"value":160},{"type":44,"tag":134,"props":283,"children":284},{"class":136,"line":163},[285,289,293],{"type":44,"tag":134,"props":286,"children":287},{"style":146},[288],{"type":49,"value":169},{"type":44,"tag":134,"props":290,"children":291},{"style":146},[292],{"type":49,"value":174},{"type":44,"tag":134,"props":294,"children":295},{"style":157},[296],{"type":49,"value":160},{"type":44,"tag":134,"props":298,"children":299},{"class":136,"line":181},[300,304,308],{"type":44,"tag":134,"props":301,"children":302},{"style":146},[303],{"type":49,"value":187},{"type":44,"tag":134,"props":305,"children":306},{"style":190},[307],{"type":49,"value":193},{"type":44,"tag":134,"props":309,"children":310},{"style":157},[311],{"type":49,"value":160},{"type":44,"tag":134,"props":313,"children":314},{"class":136,"line":200},[315,319,323],{"type":44,"tag":134,"props":316,"children":317},{"style":146},[318],{"type":49,"value":206},{"type":44,"tag":134,"props":320,"children":321},{"style":190},[322],{"type":49,"value":211},{"type":44,"tag":134,"props":324,"children":325},{"style":157},[326],{"type":49,"value":160},{"type":44,"tag":134,"props":328,"children":329},{"class":136,"line":218},[330,334,338],{"type":44,"tag":134,"props":331,"children":332},{"style":146},[333],{"type":49,"value":224},{"type":44,"tag":134,"props":335,"children":336},{"style":146},[337],{"type":49,"value":229},{"type":44,"tag":134,"props":339,"children":340},{"style":157},[341],{"type":49,"value":160},{"type":44,"tag":134,"props":343,"children":344},{"class":136,"line":236},[345,349,354],{"type":44,"tag":134,"props":346,"children":347},{"style":146},[348],{"type":49,"value":242},{"type":44,"tag":134,"props":350,"children":351},{"style":146},[352],{"type":49,"value":353}," \u002Fv1\u002Fchat\u002Fcompletions",{"type":44,"tag":134,"props":355,"children":356},{"style":157},[357],{"type":49,"value":160},{"type":44,"tag":134,"props":359,"children":361},{"class":136,"line":360},7,[362,367],{"type":44,"tag":134,"props":363,"children":364},{"style":146},[365],{"type":49,"value":366},"  --save-result",{"type":44,"tag":134,"props":368,"children":369},{"style":157},[370],{"type":49,"value":160},{"type":44,"tag":134,"props":372,"children":374},{"class":136,"line":373},8,[375,380,385],{"type":44,"tag":134,"props":376,"children":377},{"style":146},[378],{"type":49,"value":379},"  --result-dir",{"type":44,"tag":134,"props":381,"children":382},{"style":146},[383],{"type":49,"value":384}," .\u002Fbench-results",{"type":44,"tag":134,"props":386,"children":387},{"style":157},[388],{"type":49,"value":160},{"type":44,"tag":134,"props":390,"children":392},{"class":136,"line":391},9,[393,398,404,409,414,418,423],{"type":44,"tag":134,"props":394,"children":395},{"style":146},[396],{"type":49,"value":397},"  --metadata",{"type":44,"tag":134,"props":399,"children":401},{"style":400},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[402],{"type":49,"value":403}," \"",{"type":44,"tag":134,"props":405,"children":406},{"style":146},[407],{"type":49,"value":408},"version=0.6.0",{"type":44,"tag":134,"props":410,"children":411},{"style":400},[412],{"type":49,"value":413},"\"",{"type":44,"tag":134,"props":415,"children":416},{"style":400},[417],{"type":49,"value":403},{"type":44,"tag":134,"props":419,"children":420},{"style":146},[421],{"type":49,"value":422},"tp=1",{"type":44,"tag":134,"props":424,"children":425},{"style":400},[426],{"type":49,"value":427},"\"\n",{"type":44,"tag":429,"props":430,"children":431},"blockquote",{},[432],{"type":44,"tag":52,"props":433,"children":434},{},[435,440,442,448,450,456,458,464],{"type":44,"tag":117,"props":436,"children":437},{},[438],{"type":49,"value":439},"Note:",{"type":49,"value":441}," When using ",{"type":44,"tag":58,"props":443,"children":445},{"className":444},[],[446],{"type":49,"value":447},"--backend openai-chat",{"type":49,"value":449},", you must specify ",{"type":44,"tag":58,"props":451,"children":453},{"className":452},[],[454],{"type":49,"value":455},"--endpoint \u002Fv1\u002Fchat\u002Fcompletions",{"type":49,"value":457}," (default is ",{"type":44,"tag":58,"props":459,"children":461},{"className":460},[],[462],{"type":49,"value":463},"\u002Fv1\u002Fcompletions",{"type":49,"value":465},").",{"type":44,"tag":81,"props":467,"children":469},{"id":468},"core-arguments",[470],{"type":49,"value":471},"Core Arguments",{"type":44,"tag":473,"props":474,"children":475},"table",{},[476,500],{"type":44,"tag":477,"props":478,"children":479},"thead",{},[480],{"type":44,"tag":481,"props":482,"children":483},"tr",{},[484,490,495],{"type":44,"tag":485,"props":486,"children":487},"th",{},[488],{"type":49,"value":489},"Argument",{"type":44,"tag":485,"props":491,"children":492},{},[493],{"type":49,"value":494},"Default",{"type":44,"tag":485,"props":496,"children":497},{},[498],{"type":49,"value":499},"Description",{"type":44,"tag":501,"props":502,"children":503},"tbody",{},[504,573,599,625,647,680,702,728,761,782],{"type":44,"tag":481,"props":505,"children":506},{},[507,517,526],{"type":44,"tag":508,"props":509,"children":510},"td",{},[511],{"type":44,"tag":58,"props":512,"children":514},{"className":513},[],[515],{"type":49,"value":516},"--backend",{"type":44,"tag":508,"props":518,"children":519},{},[520],{"type":44,"tag":58,"props":521,"children":523},{"className":522},[],[524],{"type":49,"value":525},"openai",{"type":44,"tag":508,"props":527,"children":528},{},[529,531,536,538,544,545,551,552,557,558,564,565,571],{"type":49,"value":530},"Backend type: ",{"type":44,"tag":58,"props":532,"children":534},{"className":533},[],[535],{"type":49,"value":525},{"type":49,"value":537},", ",{"type":44,"tag":58,"props":539,"children":541},{"className":540},[],[542],{"type":49,"value":543},"openai-chat",{"type":49,"value":537},{"type":44,"tag":58,"props":546,"children":548},{"className":547},[],[549],{"type":49,"value":550},"openai-embeddings",{"type":49,"value":537},{"type":44,"tag":58,"props":553,"children":555},{"className":554},[],[556],{"type":49,"value":8},{"type":49,"value":537},{"type":44,"tag":58,"props":559,"children":561},{"className":560},[],[562],{"type":49,"value":563},"vllm-pooling",{"type":49,"value":537},{"type":44,"tag":58,"props":566,"children":568},{"className":567},[],[569],{"type":49,"value":570},"vllm-rerank",{"type":49,"value":572},", etc.",{"type":44,"tag":481,"props":574,"children":575},{},[576,585,594],{"type":44,"tag":508,"props":577,"children":578},{},[579],{"type":44,"tag":58,"props":580,"children":582},{"className":581},[],[583],{"type":49,"value":584},"--host",{"type":44,"tag":508,"props":586,"children":587},{},[588],{"type":44,"tag":58,"props":589,"children":591},{"className":590},[],[592],{"type":49,"value":593},"127.0.0.1",{"type":44,"tag":508,"props":595,"children":596},{},[597],{"type":49,"value":598},"Server host",{"type":44,"tag":481,"props":600,"children":601},{},[602,611,620],{"type":44,"tag":508,"props":603,"children":604},{},[605],{"type":44,"tag":58,"props":606,"children":608},{"className":607},[],[609],{"type":49,"value":610},"--port",{"type":44,"tag":508,"props":612,"children":613},{},[614],{"type":44,"tag":58,"props":615,"children":617},{"className":616},[],[618],{"type":49,"value":619},"8000",{"type":44,"tag":508,"props":621,"children":622},{},[623],{"type":49,"value":624},"Server port",{"type":44,"tag":481,"props":626,"children":627},{},[628,637,642],{"type":44,"tag":508,"props":629,"children":630},{},[631],{"type":44,"tag":58,"props":632,"children":634},{"className":633},[],[635],{"type":49,"value":636},"--base-url",{"type":44,"tag":508,"props":638,"children":639},{},[640],{"type":49,"value":641},"-",{"type":44,"tag":508,"props":643,"children":644},{},[645],{"type":49,"value":646},"Alternative: full base URL instead of host:port",{"type":44,"tag":481,"props":648,"children":649},{},[650,659,667],{"type":44,"tag":508,"props":651,"children":652},{},[653],{"type":44,"tag":58,"props":654,"children":656},{"className":655},[],[657],{"type":49,"value":658},"--endpoint",{"type":44,"tag":508,"props":660,"children":661},{},[662],{"type":44,"tag":58,"props":663,"children":665},{"className":664},[],[666],{"type":49,"value":463},{"type":44,"tag":508,"props":668,"children":669},{},[670,672,678],{"type":49,"value":671},"API endpoint; use ",{"type":44,"tag":58,"props":673,"children":675},{"className":674},[],[676],{"type":49,"value":677},"\u002Fv1\u002Fchat\u002Fcompletions",{"type":49,"value":679}," for openai-chat",{"type":44,"tag":481,"props":681,"children":682},{},[683,692,697],{"type":44,"tag":508,"props":684,"children":685},{},[686],{"type":44,"tag":58,"props":687,"children":689},{"className":688},[],[690],{"type":49,"value":691},"--model",{"type":44,"tag":508,"props":693,"children":694},{},[695],{"type":49,"value":696},"(from \u002Fv1\u002Fmodels)",{"type":44,"tag":508,"props":698,"children":699},{},[700],{"type":49,"value":701},"Model name",{"type":44,"tag":481,"props":703,"children":704},{},[705,714,723],{"type":44,"tag":508,"props":706,"children":707},{},[708],{"type":44,"tag":58,"props":709,"children":711},{"className":710},[],[712],{"type":49,"value":713},"--num-prompts",{"type":44,"tag":508,"props":715,"children":716},{},[717],{"type":44,"tag":58,"props":718,"children":720},{"className":719},[],[721],{"type":49,"value":722},"1000",{"type":44,"tag":508,"props":724,"children":725},{},[726],{"type":49,"value":727},"Number of prompts to process",{"type":44,"tag":481,"props":729,"children":730},{},[731,740,749],{"type":44,"tag":508,"props":732,"children":733},{},[734],{"type":44,"tag":58,"props":735,"children":737},{"className":736},[],[738],{"type":49,"value":739},"--request-rate",{"type":44,"tag":508,"props":741,"children":742},{},[743],{"type":44,"tag":58,"props":744,"children":746},{"className":745},[],[747],{"type":49,"value":748},"inf",{"type":44,"tag":508,"props":750,"children":751},{},[752,754,759],{"type":49,"value":753},"Requests per second; ",{"type":44,"tag":58,"props":755,"children":757},{"className":756},[],[758],{"type":49,"value":748},{"type":49,"value":760}," = burst all at once",{"type":44,"tag":481,"props":762,"children":763},{},[764,773,777],{"type":44,"tag":508,"props":765,"children":766},{},[767],{"type":44,"tag":58,"props":768,"children":770},{"className":769},[],[771],{"type":49,"value":772},"--max-concurrency",{"type":44,"tag":508,"props":774,"children":775},{},[776],{"type":49,"value":641},{"type":44,"tag":508,"props":778,"children":779},{},[780],{"type":49,"value":781},"Max concurrent requests (caps parallelism)",{"type":44,"tag":481,"props":783,"children":784},{},[785,794,803],{"type":44,"tag":508,"props":786,"children":787},{},[788],{"type":44,"tag":58,"props":789,"children":791},{"className":790},[],[792],{"type":49,"value":793},"--num-warmups",{"type":44,"tag":508,"props":795,"children":796},{},[797],{"type":44,"tag":58,"props":798,"children":800},{"className":799},[],[801],{"type":49,"value":802},"0",{"type":44,"tag":508,"props":804,"children":805},{},[806],{"type":49,"value":807},"Warmup requests before measuring",{"type":44,"tag":81,"props":809,"children":811},{"id":810},"datasets",[812],{"type":49,"value":813},"Datasets",{"type":44,"tag":473,"props":815,"children":816},{},[817,837],{"type":44,"tag":477,"props":818,"children":819},{},[820],{"type":44,"tag":481,"props":821,"children":822},{},[823,832],{"type":44,"tag":485,"props":824,"children":825},{},[826],{"type":44,"tag":58,"props":827,"children":829},{"className":828},[],[830],{"type":49,"value":831},"--dataset-name",{"type":44,"tag":485,"props":833,"children":834},{},[835],{"type":49,"value":836},"Use Case",{"type":44,"tag":501,"props":838,"children":839},{},[840,857,880,897,921,951,968,985],{"type":44,"tag":481,"props":841,"children":842},{},[843,852],{"type":44,"tag":508,"props":844,"children":845},{},[846],{"type":44,"tag":58,"props":847,"children":849},{"className":848},[],[850],{"type":49,"value":851},"random",{"type":44,"tag":508,"props":853,"children":854},{},[855],{"type":49,"value":856},"Synthetic random prompts (default)",{"type":44,"tag":481,"props":858,"children":859},{},[860,869],{"type":44,"tag":508,"props":861,"children":862},{},[863],{"type":44,"tag":58,"props":864,"children":866},{"className":865},[],[867],{"type":49,"value":868},"sharegpt",{"type":44,"tag":508,"props":870,"children":871},{},[872,874],{"type":49,"value":873},"ShareGPT conversation format; requires ",{"type":44,"tag":58,"props":875,"children":877},{"className":876},[],[878],{"type":49,"value":879},"--dataset-path",{"type":44,"tag":481,"props":881,"children":882},{},[883,892],{"type":44,"tag":508,"props":884,"children":885},{},[886],{"type":44,"tag":58,"props":887,"children":889},{"className":888},[],[890],{"type":49,"value":891},"sonnet",{"type":44,"tag":508,"props":893,"children":894},{},[895],{"type":49,"value":896},"Sonnet-style prompts",{"type":44,"tag":481,"props":898,"children":899},{},[900,909],{"type":44,"tag":508,"props":901,"children":902},{},[903],{"type":44,"tag":58,"props":904,"children":906},{"className":905},[],[907],{"type":49,"value":908},"hf",{"type":44,"tag":508,"props":910,"children":911},{},[912,914,919],{"type":49,"value":913},"HuggingFace dataset; requires ",{"type":44,"tag":58,"props":915,"children":917},{"className":916},[],[918],{"type":49,"value":879},{"type":49,"value":920}," (dataset ID)",{"type":44,"tag":481,"props":922,"children":923},{},[924,941],{"type":44,"tag":508,"props":925,"children":926},{},[927,933,935],{"type":44,"tag":58,"props":928,"children":930},{"className":929},[],[931],{"type":49,"value":932},"custom",{"type":49,"value":934}," \u002F ",{"type":44,"tag":58,"props":936,"children":938},{"className":937},[],[939],{"type":49,"value":940},"custom_mm",{"type":44,"tag":508,"props":942,"children":943},{},[944,946],{"type":49,"value":945},"Custom dataset; requires ",{"type":44,"tag":58,"props":947,"children":949},{"className":948},[],[950],{"type":49,"value":879},{"type":44,"tag":481,"props":952,"children":953},{},[954,963],{"type":44,"tag":508,"props":955,"children":956},{},[957],{"type":44,"tag":58,"props":958,"children":960},{"className":959},[],[961],{"type":49,"value":962},"prefix_repetition",{"type":44,"tag":508,"props":964,"children":965},{},[966],{"type":49,"value":967},"Prefix repetition benchmark",{"type":44,"tag":481,"props":969,"children":970},{},[971,980],{"type":44,"tag":508,"props":972,"children":973},{},[974],{"type":44,"tag":58,"props":975,"children":977},{"className":976},[],[978],{"type":49,"value":979},"random-mm",{"type":44,"tag":508,"props":981,"children":982},{},[983],{"type":49,"value":984},"Random multimodal (images\u002Fvideos)",{"type":44,"tag":481,"props":986,"children":987},{},[988,997],{"type":44,"tag":508,"props":989,"children":990},{},[991],{"type":44,"tag":58,"props":992,"children":994},{"className":993},[],[995],{"type":49,"value":996},"spec_bench",{"type":44,"tag":508,"props":998,"children":999},{},[1000],{"type":49,"value":1001},"Spec bench dataset",{"type":44,"tag":52,"props":1003,"children":1004},{},[1005],{"type":44,"tag":117,"props":1006,"children":1007},{},[1008],{"type":49,"value":1009},"Dataset-specific options (examples):",{"type":44,"tag":123,"props":1011,"children":1013},{"className":125,"code":1012,"language":127,"meta":128,"style":128},"# Random: control input\u002Foutput length\n--dataset-name random --random-input-len 1024 --random-output-len 128\n\n# Sonnet defaults: input 550, output 150, prefix 200\n--dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150\n\n# HuggingFace dataset\n--dataset-name hf --dataset-path \"lmarena-ai\u002FVisionArena-Chat\" --hf-split test\n\n# General overrides (map to dataset-specific args)\n--input-len 512 --output-len 256\n",[1014],{"type":44,"tag":58,"props":1015,"children":1016},{"__ignoreMap":128},[1017,1026,1058,1067,1075,1107,1114,1122,1162,1169,1178],{"type":44,"tag":134,"props":1018,"children":1019},{"class":136,"line":137},[1020],{"type":44,"tag":134,"props":1021,"children":1023},{"style":1022},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1024],{"type":49,"value":1025},"# Random: control input\u002Foutput length\n",{"type":44,"tag":134,"props":1027,"children":1028},{"class":136,"line":163},[1029,1033,1038,1043,1048,1053],{"type":44,"tag":134,"props":1030,"children":1031},{"style":141},[1032],{"type":49,"value":831},{"type":44,"tag":134,"props":1034,"children":1035},{"style":146},[1036],{"type":49,"value":1037}," random",{"type":44,"tag":134,"props":1039,"children":1040},{"style":146},[1041],{"type":49,"value":1042}," --random-input-len",{"type":44,"tag":134,"props":1044,"children":1045},{"style":190},[1046],{"type":49,"value":1047}," 1024",{"type":44,"tag":134,"props":1049,"children":1050},{"style":146},[1051],{"type":49,"value":1052}," --random-output-len",{"type":44,"tag":134,"props":1054,"children":1055},{"style":190},[1056],{"type":49,"value":1057}," 128\n",{"type":44,"tag":134,"props":1059,"children":1060},{"class":136,"line":181},[1061],{"type":44,"tag":134,"props":1062,"children":1064},{"emptyLinePlaceholder":1063},true,[1065],{"type":49,"value":1066},"\n",{"type":44,"tag":134,"props":1068,"children":1069},{"class":136,"line":200},[1070],{"type":44,"tag":134,"props":1071,"children":1072},{"style":1022},[1073],{"type":49,"value":1074},"# Sonnet defaults: input 550, output 150, prefix 200\n",{"type":44,"tag":134,"props":1076,"children":1077},{"class":136,"line":218},[1078,1082,1087,1092,1097,1102],{"type":44,"tag":134,"props":1079,"children":1080},{"style":141},[1081],{"type":49,"value":831},{"type":44,"tag":134,"props":1083,"children":1084},{"style":146},[1085],{"type":49,"value":1086}," sonnet",{"type":44,"tag":134,"props":1088,"children":1089},{"style":146},[1090],{"type":49,"value":1091}," --sonnet-input-len",{"type":44,"tag":134,"props":1093,"children":1094},{"style":190},[1095],{"type":49,"value":1096}," 550",{"type":44,"tag":134,"props":1098,"children":1099},{"style":146},[1100],{"type":49,"value":1101}," --sonnet-output-len",{"type":44,"tag":134,"props":1103,"children":1104},{"style":190},[1105],{"type":49,"value":1106}," 150\n",{"type":44,"tag":134,"props":1108,"children":1109},{"class":136,"line":236},[1110],{"type":44,"tag":134,"props":1111,"children":1112},{"emptyLinePlaceholder":1063},[1113],{"type":49,"value":1066},{"type":44,"tag":134,"props":1115,"children":1116},{"class":136,"line":360},[1117],{"type":44,"tag":134,"props":1118,"children":1119},{"style":1022},[1120],{"type":49,"value":1121},"# HuggingFace dataset\n",{"type":44,"tag":134,"props":1123,"children":1124},{"class":136,"line":373},[1125,1129,1134,1139,1143,1148,1152,1157],{"type":44,"tag":134,"props":1126,"children":1127},{"style":141},[1128],{"type":49,"value":831},{"type":44,"tag":134,"props":1130,"children":1131},{"style":146},[1132],{"type":49,"value":1133}," hf",{"type":44,"tag":134,"props":1135,"children":1136},{"style":146},[1137],{"type":49,"value":1138}," --dataset-path",{"type":44,"tag":134,"props":1140,"children":1141},{"style":400},[1142],{"type":49,"value":403},{"type":44,"tag":134,"props":1144,"children":1145},{"style":146},[1146],{"type":49,"value":1147},"lmarena-ai\u002FVisionArena-Chat",{"type":44,"tag":134,"props":1149,"children":1150},{"style":400},[1151],{"type":49,"value":413},{"type":44,"tag":134,"props":1153,"children":1154},{"style":146},[1155],{"type":49,"value":1156}," --hf-split",{"type":44,"tag":134,"props":1158,"children":1159},{"style":146},[1160],{"type":49,"value":1161}," test\n",{"type":44,"tag":134,"props":1163,"children":1164},{"class":136,"line":391},[1165],{"type":44,"tag":134,"props":1166,"children":1167},{"emptyLinePlaceholder":1063},[1168],{"type":49,"value":1066},{"type":44,"tag":134,"props":1170,"children":1172},{"class":136,"line":1171},10,[1173],{"type":44,"tag":134,"props":1174,"children":1175},{"style":1022},[1176],{"type":49,"value":1177},"# General overrides (map to dataset-specific args)\n",{"type":44,"tag":134,"props":1179,"children":1181},{"class":136,"line":1180},11,[1182,1187,1192,1197],{"type":44,"tag":134,"props":1183,"children":1184},{"style":141},[1185],{"type":49,"value":1186},"--input-len",{"type":44,"tag":134,"props":1188,"children":1189},{"style":190},[1190],{"type":49,"value":1191}," 512",{"type":44,"tag":134,"props":1193,"children":1194},{"style":146},[1195],{"type":49,"value":1196}," --output-len",{"type":44,"tag":134,"props":1198,"children":1199},{"style":190},[1200],{"type":49,"value":1201}," 256\n",{"type":44,"tag":81,"props":1203,"children":1205},{"id":1204},"load-control",[1206],{"type":49,"value":1207},"Load Control",{"type":44,"tag":123,"props":1209,"children":1211},{"className":125,"code":1210,"language":127,"meta":128,"style":128},"# Fixed request rate (Poisson process)\n--request-rate 10\n\n# More bursty arrivals (gamma distribution, burstiness \u003C 1)\n--request-rate 10 --burstiness 0.5\n\n# Ramp-up from low to high RPS\n--ramp-up-strategy linear --ramp-up-start-rps 1 --ramp-up-end-rps 50\n\n# Limit concurrency (useful for rate-limited APIs)\n--max-concurrency 32\n",[1212],{"type":44,"tag":58,"props":1213,"children":1214},{"__ignoreMap":128},[1215,1223,1235,1242,1250,1272,1279,1287,1320,1327,1335],{"type":44,"tag":134,"props":1216,"children":1217},{"class":136,"line":137},[1218],{"type":44,"tag":134,"props":1219,"children":1220},{"style":1022},[1221],{"type":49,"value":1222},"# Fixed request rate (Poisson process)\n",{"type":44,"tag":134,"props":1224,"children":1225},{"class":136,"line":163},[1226,1230],{"type":44,"tag":134,"props":1227,"children":1228},{"style":141},[1229],{"type":49,"value":739},{"type":44,"tag":134,"props":1231,"children":1232},{"style":190},[1233],{"type":49,"value":1234}," 10\n",{"type":44,"tag":134,"props":1236,"children":1237},{"class":136,"line":181},[1238],{"type":44,"tag":134,"props":1239,"children":1240},{"emptyLinePlaceholder":1063},[1241],{"type":49,"value":1066},{"type":44,"tag":134,"props":1243,"children":1244},{"class":136,"line":200},[1245],{"type":44,"tag":134,"props":1246,"children":1247},{"style":1022},[1248],{"type":49,"value":1249},"# More bursty arrivals (gamma distribution, burstiness \u003C 1)\n",{"type":44,"tag":134,"props":1251,"children":1252},{"class":136,"line":218},[1253,1257,1262,1267],{"type":44,"tag":134,"props":1254,"children":1255},{"style":141},[1256],{"type":49,"value":739},{"type":44,"tag":134,"props":1258,"children":1259},{"style":190},[1260],{"type":49,"value":1261}," 10",{"type":44,"tag":134,"props":1263,"children":1264},{"style":146},[1265],{"type":49,"value":1266}," --burstiness",{"type":44,"tag":134,"props":1268,"children":1269},{"style":190},[1270],{"type":49,"value":1271}," 0.5\n",{"type":44,"tag":134,"props":1273,"children":1274},{"class":136,"line":236},[1275],{"type":44,"tag":134,"props":1276,"children":1277},{"emptyLinePlaceholder":1063},[1278],{"type":49,"value":1066},{"type":44,"tag":134,"props":1280,"children":1281},{"class":136,"line":360},[1282],{"type":44,"tag":134,"props":1283,"children":1284},{"style":1022},[1285],{"type":49,"value":1286},"# Ramp-up from low to high RPS\n",{"type":44,"tag":134,"props":1288,"children":1289},{"class":136,"line":373},[1290,1295,1300,1305,1310,1315],{"type":44,"tag":134,"props":1291,"children":1292},{"style":141},[1293],{"type":49,"value":1294},"--ramp-up-strategy",{"type":44,"tag":134,"props":1296,"children":1297},{"style":146},[1298],{"type":49,"value":1299}," linear",{"type":44,"tag":134,"props":1301,"children":1302},{"style":146},[1303],{"type":49,"value":1304}," --ramp-up-start-rps",{"type":44,"tag":134,"props":1306,"children":1307},{"style":190},[1308],{"type":49,"value":1309}," 1",{"type":44,"tag":134,"props":1311,"children":1312},{"style":146},[1313],{"type":49,"value":1314}," --ramp-up-end-rps",{"type":44,"tag":134,"props":1316,"children":1317},{"style":190},[1318],{"type":49,"value":1319}," 50\n",{"type":44,"tag":134,"props":1321,"children":1322},{"class":136,"line":391},[1323],{"type":44,"tag":134,"props":1324,"children":1325},{"emptyLinePlaceholder":1063},[1326],{"type":49,"value":1066},{"type":44,"tag":134,"props":1328,"children":1329},{"class":136,"line":1171},[1330],{"type":44,"tag":134,"props":1331,"children":1332},{"style":1022},[1333],{"type":49,"value":1334},"# Limit concurrency (useful for rate-limited APIs)\n",{"type":44,"tag":134,"props":1336,"children":1337},{"class":136,"line":1180},[1338,1342],{"type":44,"tag":134,"props":1339,"children":1340},{"style":141},[1341],{"type":49,"value":772},{"type":44,"tag":134,"props":1343,"children":1344},{"style":190},[1345],{"type":49,"value":1346}," 32\n",{"type":44,"tag":81,"props":1348,"children":1350},{"id":1349},"results-and-metrics",[1351],{"type":49,"value":1352},"Results and Metrics",{"type":44,"tag":473,"props":1354,"children":1355},{},[1356,1370],{"type":44,"tag":477,"props":1357,"children":1358},{},[1359],{"type":44,"tag":481,"props":1360,"children":1361},{},[1362,1366],{"type":44,"tag":485,"props":1363,"children":1364},{},[1365],{"type":49,"value":489},{"type":44,"tag":485,"props":1367,"children":1368},{},[1369],{"type":49,"value":499},{"type":44,"tag":501,"props":1371,"children":1372},{},[1373,1390,1407,1424,1441,1466,1519,1550],{"type":44,"tag":481,"props":1374,"children":1375},{},[1376,1385],{"type":44,"tag":508,"props":1377,"children":1378},{},[1379],{"type":44,"tag":58,"props":1380,"children":1382},{"className":1381},[],[1383],{"type":49,"value":1384},"--save-result",{"type":44,"tag":508,"props":1386,"children":1387},{},[1388],{"type":49,"value":1389},"Save benchmark results to JSON",{"type":44,"tag":481,"props":1391,"children":1392},{},[1393,1402],{"type":44,"tag":508,"props":1394,"children":1395},{},[1396],{"type":44,"tag":58,"props":1397,"children":1399},{"className":1398},[],[1400],{"type":49,"value":1401},"--save-detailed",{"type":44,"tag":508,"props":1403,"children":1404},{},[1405],{"type":49,"value":1406},"Include per-request TTFT, TPOT, errors in JSON",{"type":44,"tag":481,"props":1408,"children":1409},{},[1410,1419],{"type":44,"tag":508,"props":1411,"children":1412},{},[1413],{"type":44,"tag":58,"props":1414,"children":1416},{"className":1415},[],[1417],{"type":49,"value":1418},"--append-result",{"type":44,"tag":508,"props":1420,"children":1421},{},[1422],{"type":49,"value":1423},"Append to existing result file",{"type":44,"tag":481,"props":1425,"children":1426},{},[1427,1436],{"type":44,"tag":508,"props":1428,"children":1429},{},[1430],{"type":44,"tag":58,"props":1431,"children":1433},{"className":1432},[],[1434],{"type":49,"value":1435},"--result-dir",{"type":44,"tag":508,"props":1437,"children":1438},{},[1439],{"type":49,"value":1440},"Directory for result files",{"type":44,"tag":481,"props":1442,"children":1443},{},[1444,1453],{"type":44,"tag":508,"props":1445,"children":1446},{},[1447],{"type":44,"tag":58,"props":1448,"children":1450},{"className":1449},[],[1451],{"type":49,"value":1452},"--result-filename",{"type":44,"tag":508,"props":1454,"children":1455},{},[1456,1458,1464],{"type":49,"value":1457},"Custom filename (default: ",{"type":44,"tag":58,"props":1459,"children":1461},{"className":1460},[],[1462],{"type":49,"value":1463},"{label}-{request_rate}qps-{model}-{timestamp}.json",{"type":49,"value":1465},")",{"type":44,"tag":481,"props":1467,"children":1468},{},[1469,1478],{"type":44,"tag":508,"props":1470,"children":1471},{},[1472],{"type":44,"tag":58,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":49,"value":1477},"--percentile-metrics",{"type":44,"tag":508,"props":1479,"children":1480},{},[1481,1483,1489,1490,1496,1497,1503,1504,1510,1512,1518],{"type":49,"value":1482},"Metrics for percentiles: ",{"type":44,"tag":58,"props":1484,"children":1486},{"className":1485},[],[1487],{"type":49,"value":1488},"ttft",{"type":49,"value":537},{"type":44,"tag":58,"props":1491,"children":1493},{"className":1492},[],[1494],{"type":49,"value":1495},"tpot",{"type":49,"value":537},{"type":44,"tag":58,"props":1498,"children":1500},{"className":1499},[],[1501],{"type":49,"value":1502},"itl",{"type":49,"value":537},{"type":44,"tag":58,"props":1505,"children":1507},{"className":1506},[],[1508],{"type":49,"value":1509},"e2el",{"type":49,"value":1511}," (default: ",{"type":44,"tag":58,"props":1513,"children":1515},{"className":1514},[],[1516],{"type":49,"value":1517},"ttft,tpot,itl",{"type":49,"value":1465},{"type":44,"tag":481,"props":1520,"children":1521},{},[1522,1531],{"type":44,"tag":508,"props":1523,"children":1524},{},[1525],{"type":44,"tag":58,"props":1526,"children":1528},{"className":1527},[],[1529],{"type":49,"value":1530},"--metric-percentiles",{"type":44,"tag":508,"props":1532,"children":1533},{},[1534,1536,1542,1543,1549],{"type":49,"value":1535},"Percentile values, e.g. ",{"type":44,"tag":58,"props":1537,"children":1539},{"className":1538},[],[1540],{"type":49,"value":1541},"25,50,99",{"type":49,"value":1511},{"type":44,"tag":58,"props":1544,"children":1546},{"className":1545},[],[1547],{"type":49,"value":1548},"99",{"type":49,"value":1465},{"type":44,"tag":481,"props":1551,"children":1552},{},[1553,1562],{"type":44,"tag":508,"props":1554,"children":1555},{},[1556],{"type":44,"tag":58,"props":1557,"children":1559},{"className":1558},[],[1560],{"type":49,"value":1561},"--goodput",{"type":44,"tag":508,"props":1563,"children":1564},{},[1565,1567,1573],{"type":49,"value":1566},"SLO for goodput: ",{"type":44,"tag":58,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":49,"value":1572},"ttft:500 tpot:50",{"type":49,"value":1574}," (ms)",{"type":44,"tag":81,"props":1576,"children":1578},{"id":1577},"sampling-parameters-openai-compatible-backends",[1579],{"type":49,"value":1580},"Sampling Parameters (OpenAI-compatible backends)",{"type":44,"tag":123,"props":1582,"children":1584},{"className":125,"code":1583,"language":127,"meta":128,"style":128},"--temperature 0.7 --top-p 0.95 --top-k 50\n--frequency-penalty 0 --presence-penalty 0 --repetition-penalty 1.0\n",[1585],{"type":44,"tag":58,"props":1586,"children":1587},{"__ignoreMap":128},[1588,1620],{"type":44,"tag":134,"props":1589,"children":1590},{"class":136,"line":137},[1591,1596,1601,1606,1611,1616],{"type":44,"tag":134,"props":1592,"children":1593},{"style":141},[1594],{"type":49,"value":1595},"--temperature",{"type":44,"tag":134,"props":1597,"children":1598},{"style":190},[1599],{"type":49,"value":1600}," 0.7",{"type":44,"tag":134,"props":1602,"children":1603},{"style":146},[1604],{"type":49,"value":1605}," --top-p",{"type":44,"tag":134,"props":1607,"children":1608},{"style":190},[1609],{"type":49,"value":1610}," 0.95",{"type":44,"tag":134,"props":1612,"children":1613},{"style":146},[1614],{"type":49,"value":1615}," --top-k",{"type":44,"tag":134,"props":1617,"children":1618},{"style":190},[1619],{"type":49,"value":1319},{"type":44,"tag":134,"props":1621,"children":1622},{"class":136,"line":163},[1623,1628,1633,1638,1642,1647],{"type":44,"tag":134,"props":1624,"children":1625},{"style":141},[1626],{"type":49,"value":1627},"--frequency-penalty",{"type":44,"tag":134,"props":1629,"children":1630},{"style":190},[1631],{"type":49,"value":1632}," 0",{"type":44,"tag":134,"props":1634,"children":1635},{"style":146},[1636],{"type":49,"value":1637}," --presence-penalty",{"type":44,"tag":134,"props":1639,"children":1640},{"style":190},[1641],{"type":49,"value":1632},{"type":44,"tag":134,"props":1643,"children":1644},{"style":146},[1645],{"type":49,"value":1646}," --repetition-penalty",{"type":44,"tag":134,"props":1648,"children":1649},{"style":190},[1650],{"type":49,"value":1651}," 1.0\n",{"type":44,"tag":81,"props":1653,"children":1655},{"id":1654},"common-workflows",[1656],{"type":49,"value":1657},"Common Workflows",{"type":44,"tag":52,"props":1659,"children":1660},{},[1661],{"type":44,"tag":117,"props":1662,"children":1663},{},[1664],{"type":49,"value":1665},"1. Throughput test with random dataset (burst):",{"type":44,"tag":123,"props":1667,"children":1669},{"className":125,"code":1668,"language":127,"meta":128,"style":128},"vllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --dataset-name random \\\n  --num-prompts 500 --random-input-len 512 --random-output-len 128\n",[1670],{"type":44,"tag":58,"props":1671,"children":1672},{"__ignoreMap":128},[1673,1719,1734,1749,1765],{"type":44,"tag":134,"props":1674,"children":1675},{"class":136,"line":137},[1676,1680,1684,1688,1693,1697,1702,1706,1711,1715],{"type":44,"tag":134,"props":1677,"children":1678},{"style":141},[1679],{"type":49,"value":8},{"type":44,"tag":134,"props":1681,"children":1682},{"style":146},[1683],{"type":49,"value":149},{"type":44,"tag":134,"props":1685,"children":1686},{"style":146},[1687],{"type":49,"value":154},{"type":44,"tag":134,"props":1689,"children":1690},{"style":146},[1691],{"type":49,"value":1692}," --backend",{"type":44,"tag":134,"props":1694,"children":1695},{"style":146},[1696],{"type":49,"value":174},{"type":44,"tag":134,"props":1698,"children":1699},{"style":146},[1700],{"type":49,"value":1701}," --host",{"type":44,"tag":134,"props":1703,"children":1704},{"style":190},[1705],{"type":49,"value":193},{"type":44,"tag":134,"props":1707,"children":1708},{"style":146},[1709],{"type":49,"value":1710}," --port",{"type":44,"tag":134,"props":1712,"children":1713},{"style":190},[1714],{"type":49,"value":211},{"type":44,"tag":134,"props":1716,"children":1717},{"style":157},[1718],{"type":49,"value":160},{"type":44,"tag":134,"props":1720,"children":1721},{"class":136,"line":163},[1722,1726,1730],{"type":44,"tag":134,"props":1723,"children":1724},{"style":146},[1725],{"type":49,"value":224},{"type":44,"tag":134,"props":1727,"children":1728},{"style":146},[1729],{"type":49,"value":229},{"type":44,"tag":134,"props":1731,"children":1732},{"style":157},[1733],{"type":49,"value":160},{"type":44,"tag":134,"props":1735,"children":1736},{"class":136,"line":181},[1737,1741,1745],{"type":44,"tag":134,"props":1738,"children":1739},{"style":146},[1740],{"type":49,"value":242},{"type":44,"tag":134,"props":1742,"children":1743},{"style":146},[1744],{"type":49,"value":353},{"type":44,"tag":134,"props":1746,"children":1747},{"style":157},[1748],{"type":49,"value":160},{"type":44,"tag":134,"props":1750,"children":1751},{"class":136,"line":200},[1752,1757,1761],{"type":44,"tag":134,"props":1753,"children":1754},{"style":146},[1755],{"type":49,"value":1756},"  --dataset-name",{"type":44,"tag":134,"props":1758,"children":1759},{"style":146},[1760],{"type":49,"value":1037},{"type":44,"tag":134,"props":1762,"children":1763},{"style":157},[1764],{"type":49,"value":160},{"type":44,"tag":134,"props":1766,"children":1767},{"class":136,"line":218},[1768,1773,1778,1782,1786,1790],{"type":44,"tag":134,"props":1769,"children":1770},{"style":146},[1771],{"type":49,"value":1772},"  --num-prompts",{"type":44,"tag":134,"props":1774,"children":1775},{"style":190},[1776],{"type":49,"value":1777}," 500",{"type":44,"tag":134,"props":1779,"children":1780},{"style":146},[1781],{"type":49,"value":1042},{"type":44,"tag":134,"props":1783,"children":1784},{"style":190},[1785],{"type":49,"value":1191},{"type":44,"tag":134,"props":1787,"children":1788},{"style":146},[1789],{"type":49,"value":1052},{"type":44,"tag":134,"props":1791,"children":1792},{"style":190},[1793],{"type":49,"value":1057},{"type":44,"tag":52,"props":1795,"children":1796},{},[1797],{"type":44,"tag":117,"props":1798,"children":1799},{},[1800],{"type":49,"value":1801},"2. Latency test with fixed QPS:",{"type":44,"tag":123,"props":1803,"children":1805},{"className":125,"code":1804,"language":127,"meta":128,"style":128},"vllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --request-rate 5 --num-prompts 200 \\\n  --save-result --percentile-metrics ttft,tpot --metric-percentiles 50,99\n",[1806],{"type":44,"tag":58,"props":1807,"children":1808},{"__ignoreMap":128},[1809,1852,1867,1882,1909],{"type":44,"tag":134,"props":1810,"children":1811},{"class":136,"line":137},[1812,1816,1820,1824,1828,1832,1836,1840,1844,1848],{"type":44,"tag":134,"props":1813,"children":1814},{"style":141},[1815],{"type":49,"value":8},{"type":44,"tag":134,"props":1817,"children":1818},{"style":146},[1819],{"type":49,"value":149},{"type":44,"tag":134,"props":1821,"children":1822},{"style":146},[1823],{"type":49,"value":154},{"type":44,"tag":134,"props":1825,"children":1826},{"style":146},[1827],{"type":49,"value":1692},{"type":44,"tag":134,"props":1829,"children":1830},{"style":146},[1831],{"type":49,"value":174},{"type":44,"tag":134,"props":1833,"children":1834},{"style":146},[1835],{"type":49,"value":1701},{"type":44,"tag":134,"props":1837,"children":1838},{"style":190},[1839],{"type":49,"value":193},{"type":44,"tag":134,"props":1841,"children":1842},{"style":146},[1843],{"type":49,"value":1710},{"type":44,"tag":134,"props":1845,"children":1846},{"style":190},[1847],{"type":49,"value":211},{"type":44,"tag":134,"props":1849,"children":1850},{"style":157},[1851],{"type":49,"value":160},{"type":44,"tag":134,"props":1853,"children":1854},{"class":136,"line":163},[1855,1859,1863],{"type":44,"tag":134,"props":1856,"children":1857},{"style":146},[1858],{"type":49,"value":224},{"type":44,"tag":134,"props":1860,"children":1861},{"style":146},[1862],{"type":49,"value":229},{"type":44,"tag":134,"props":1864,"children":1865},{"style":157},[1866],{"type":49,"value":160},{"type":44,"tag":134,"props":1868,"children":1869},{"class":136,"line":181},[1870,1874,1878],{"type":44,"tag":134,"props":1871,"children":1872},{"style":146},[1873],{"type":49,"value":242},{"type":44,"tag":134,"props":1875,"children":1876},{"style":146},[1877],{"type":49,"value":353},{"type":44,"tag":134,"props":1879,"children":1880},{"style":157},[1881],{"type":49,"value":160},{"type":44,"tag":134,"props":1883,"children":1884},{"class":136,"line":200},[1885,1890,1895,1900,1905],{"type":44,"tag":134,"props":1886,"children":1887},{"style":146},[1888],{"type":49,"value":1889},"  --request-rate",{"type":44,"tag":134,"props":1891,"children":1892},{"style":190},[1893],{"type":49,"value":1894}," 5",{"type":44,"tag":134,"props":1896,"children":1897},{"style":146},[1898],{"type":49,"value":1899}," --num-prompts",{"type":44,"tag":134,"props":1901,"children":1902},{"style":190},[1903],{"type":49,"value":1904}," 200",{"type":44,"tag":134,"props":1906,"children":1907},{"style":157},[1908],{"type":49,"value":160},{"type":44,"tag":134,"props":1910,"children":1911},{"class":136,"line":218},[1912,1916,1921,1926,1931],{"type":44,"tag":134,"props":1913,"children":1914},{"style":146},[1915],{"type":49,"value":366},{"type":44,"tag":134,"props":1917,"children":1918},{"style":146},[1919],{"type":49,"value":1920}," --percentile-metrics",{"type":44,"tag":134,"props":1922,"children":1923},{"style":146},[1924],{"type":49,"value":1925}," ttft,tpot",{"type":44,"tag":134,"props":1927,"children":1928},{"style":146},[1929],{"type":49,"value":1930}," --metric-percentiles",{"type":44,"tag":134,"props":1932,"children":1933},{"style":146},[1934],{"type":49,"value":1935}," 50,99\n",{"type":44,"tag":52,"props":1937,"children":1938},{},[1939],{"type":44,"tag":117,"props":1940,"children":1941},{},[1942],{"type":49,"value":1943},"3. Benchmark against remote API (base-url):",{"type":44,"tag":123,"props":1945,"children":1947},{"className":125,"code":1946,"language":127,"meta":128,"style":128},"vllm bench serve --backend openai-chat \\\n  --base-url \"https:\u002F\u002Fapi.example.com\u002Fv1\" \\\n  --model my-model \\\n  --header \"Authorization=Bearer $API_KEY\"\n",[1948],{"type":44,"tag":58,"props":1949,"children":1950},{"__ignoreMap":128},[1951,1978,2003,2019],{"type":44,"tag":134,"props":1952,"children":1953},{"class":136,"line":137},[1954,1958,1962,1966,1970,1974],{"type":44,"tag":134,"props":1955,"children":1956},{"style":141},[1957],{"type":49,"value":8},{"type":44,"tag":134,"props":1959,"children":1960},{"style":146},[1961],{"type":49,"value":149},{"type":44,"tag":134,"props":1963,"children":1964},{"style":146},[1965],{"type":49,"value":154},{"type":44,"tag":134,"props":1967,"children":1968},{"style":146},[1969],{"type":49,"value":1692},{"type":44,"tag":134,"props":1971,"children":1972},{"style":146},[1973],{"type":49,"value":174},{"type":44,"tag":134,"props":1975,"children":1976},{"style":157},[1977],{"type":49,"value":160},{"type":44,"tag":134,"props":1979,"children":1980},{"class":136,"line":163},[1981,1986,1990,1995,1999],{"type":44,"tag":134,"props":1982,"children":1983},{"style":146},[1984],{"type":49,"value":1985},"  --base-url",{"type":44,"tag":134,"props":1987,"children":1988},{"style":400},[1989],{"type":49,"value":403},{"type":44,"tag":134,"props":1991,"children":1992},{"style":146},[1993],{"type":49,"value":1994},"https:\u002F\u002Fapi.example.com\u002Fv1",{"type":44,"tag":134,"props":1996,"children":1997},{"style":400},[1998],{"type":49,"value":413},{"type":44,"tag":134,"props":2000,"children":2001},{"style":157},[2002],{"type":49,"value":160},{"type":44,"tag":134,"props":2004,"children":2005},{"class":136,"line":181},[2006,2010,2015],{"type":44,"tag":134,"props":2007,"children":2008},{"style":146},[2009],{"type":49,"value":224},{"type":44,"tag":134,"props":2011,"children":2012},{"style":146},[2013],{"type":49,"value":2014}," my-model",{"type":44,"tag":134,"props":2016,"children":2017},{"style":157},[2018],{"type":49,"value":160},{"type":44,"tag":134,"props":2020,"children":2021},{"class":136,"line":200},[2022,2027,2031,2036,2041],{"type":44,"tag":134,"props":2023,"children":2024},{"style":146},[2025],{"type":49,"value":2026},"  --header",{"type":44,"tag":134,"props":2028,"children":2029},{"style":400},[2030],{"type":49,"value":403},{"type":44,"tag":134,"props":2032,"children":2033},{"style":146},[2034],{"type":49,"value":2035},"Authorization=Bearer ",{"type":44,"tag":134,"props":2037,"children":2038},{"style":157},[2039],{"type":49,"value":2040},"$API_KEY",{"type":44,"tag":134,"props":2042,"children":2043},{"style":400},[2044],{"type":49,"value":427},{"type":44,"tag":52,"props":2046,"children":2047},{},[2048],{"type":44,"tag":117,"props":2049,"children":2050},{},[2051],{"type":49,"value":2052},"4. Run inside Docker (when vLLM client not on host):",{"type":44,"tag":123,"props":2054,"children":2056},{"className":125,"code":2055,"language":127,"meta":128,"style":128},"docker exec \u003Ccontainer-name> vllm bench serve \\\n  --backend openai-chat --host 127.0.0.1 --port 8000 \\\n  --model Qwen\u002FQwen2.5-1.5B-Instruct \\\n  --endpoint \u002Fv1\u002Fchat\u002Fcompletions \\\n  --dataset-name random --num-prompts 100\n",[2057],{"type":44,"tag":58,"props":2058,"children":2059},{"__ignoreMap":128},[2060,2110,2141,2156,2171],{"type":44,"tag":134,"props":2061,"children":2062},{"class":136,"line":137},[2063,2068,2073,2078,2083,2088,2093,2098,2102,2106],{"type":44,"tag":134,"props":2064,"children":2065},{"style":141},[2066],{"type":49,"value":2067},"docker",{"type":44,"tag":134,"props":2069,"children":2070},{"style":146},[2071],{"type":49,"value":2072}," exec",{"type":44,"tag":134,"props":2074,"children":2075},{"style":400},[2076],{"type":49,"value":2077}," \u003C",{"type":44,"tag":134,"props":2079,"children":2080},{"style":146},[2081],{"type":49,"value":2082},"container-nam",{"type":44,"tag":134,"props":2084,"children":2085},{"style":157},[2086],{"type":49,"value":2087},"e",{"type":44,"tag":134,"props":2089,"children":2090},{"style":400},[2091],{"type":49,"value":2092},">",{"type":44,"tag":134,"props":2094,"children":2095},{"style":146},[2096],{"type":49,"value":2097}," vllm",{"type":44,"tag":134,"props":2099,"children":2100},{"style":146},[2101],{"type":49,"value":149},{"type":44,"tag":134,"props":2103,"children":2104},{"style":146},[2105],{"type":49,"value":154},{"type":44,"tag":134,"props":2107,"children":2108},{"style":157},[2109],{"type":49,"value":160},{"type":44,"tag":134,"props":2111,"children":2112},{"class":136,"line":163},[2113,2117,2121,2125,2129,2133,2137],{"type":44,"tag":134,"props":2114,"children":2115},{"style":146},[2116],{"type":49,"value":169},{"type":44,"tag":134,"props":2118,"children":2119},{"style":146},[2120],{"type":49,"value":174},{"type":44,"tag":134,"props":2122,"children":2123},{"style":146},[2124],{"type":49,"value":1701},{"type":44,"tag":134,"props":2126,"children":2127},{"style":190},[2128],{"type":49,"value":193},{"type":44,"tag":134,"props":2130,"children":2131},{"style":146},[2132],{"type":49,"value":1710},{"type":44,"tag":134,"props":2134,"children":2135},{"style":190},[2136],{"type":49,"value":211},{"type":44,"tag":134,"props":2138,"children":2139},{"style":157},[2140],{"type":49,"value":160},{"type":44,"tag":134,"props":2142,"children":2143},{"class":136,"line":181},[2144,2148,2152],{"type":44,"tag":134,"props":2145,"children":2146},{"style":146},[2147],{"type":49,"value":224},{"type":44,"tag":134,"props":2149,"children":2150},{"style":146},[2151],{"type":49,"value":229},{"type":44,"tag":134,"props":2153,"children":2154},{"style":157},[2155],{"type":49,"value":160},{"type":44,"tag":134,"props":2157,"children":2158},{"class":136,"line":200},[2159,2163,2167],{"type":44,"tag":134,"props":2160,"children":2161},{"style":146},[2162],{"type":49,"value":242},{"type":44,"tag":134,"props":2164,"children":2165},{"style":146},[2166],{"type":49,"value":353},{"type":44,"tag":134,"props":2168,"children":2169},{"style":157},[2170],{"type":49,"value":160},{"type":44,"tag":134,"props":2172,"children":2173},{"class":136,"line":218},[2174,2178,2182,2186],{"type":44,"tag":134,"props":2175,"children":2176},{"style":146},[2177],{"type":49,"value":1756},{"type":44,"tag":134,"props":2179,"children":2180},{"style":146},[2181],{"type":49,"value":1037},{"type":44,"tag":134,"props":2183,"children":2184},{"style":146},[2185],{"type":49,"value":1899},{"type":44,"tag":134,"props":2187,"children":2188},{"style":190},[2189],{"type":49,"value":2190}," 100\n",{"type":44,"tag":81,"props":2192,"children":2194},{"id":2193},"troubleshooting",[2195],{"type":49,"value":2196},"Troubleshooting",{"type":44,"tag":88,"props":2198,"children":2199},{},[2200,2231,2256,2280,2302,2319],{"type":44,"tag":92,"props":2201,"children":2202},{},[2203,2208,2210,2215,2217,2222,2224,2229],{"type":44,"tag":117,"props":2204,"children":2205},{},[2206],{"type":49,"value":2207},"Connection refused",{"type":49,"value":2209},": Ensure the server is running and ",{"type":44,"tag":58,"props":2211,"children":2213},{"className":2212},[],[2214],{"type":49,"value":584},{"type":49,"value":2216},"\u002F",{"type":44,"tag":58,"props":2218,"children":2220},{"className":2219},[],[2221],{"type":49,"value":610},{"type":49,"value":2223}," or ",{"type":44,"tag":58,"props":2225,"children":2227},{"className":2226},[],[2228],{"type":49,"value":636},{"type":49,"value":2230}," are correct.",{"type":44,"tag":92,"props":2232,"children":2233},{},[2234,2239,2241,2246,2248,2254],{"type":44,"tag":117,"props":2235,"children":2236},{},[2237],{"type":49,"value":2238},"Model not found",{"type":49,"value":2240},": Pass ",{"type":44,"tag":58,"props":2242,"children":2244},{"className":2243},[],[2245],{"type":49,"value":691},{"type":49,"value":2247}," explicitly or ensure ",{"type":44,"tag":58,"props":2249,"children":2251},{"className":2250},[],[2252],{"type":49,"value":2253},"\u002Fv1\u002Fmodels",{"type":49,"value":2255}," returns the model.",{"type":44,"tag":92,"props":2257,"children":2258},{},[2259,2264,2266,2271,2273,2278],{"type":44,"tag":117,"props":2260,"children":2261},{},[2262],{"type":49,"value":2263},"URL must end with chat\u002Fcompletions",{"type":49,"value":2265},": Use ",{"type":44,"tag":58,"props":2267,"children":2269},{"className":2268},[],[2270],{"type":49,"value":455},{"type":49,"value":2272}," when ",{"type":44,"tag":58,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":49,"value":447},{"type":49,"value":2279},".",{"type":44,"tag":92,"props":2281,"children":2282},{},[2283,2288,2290,2295,2296,2301],{"type":44,"tag":117,"props":2284,"children":2285},{},[2286],{"type":49,"value":2287},"Rate limit \u002F 429",{"type":49,"value":2289},": Reduce ",{"type":44,"tag":58,"props":2291,"children":2293},{"className":2292},[],[2294],{"type":49,"value":739},{"type":49,"value":2223},{"type":44,"tag":58,"props":2297,"children":2299},{"className":2298},[],[2300],{"type":49,"value":772},{"type":49,"value":2279},{"type":44,"tag":92,"props":2303,"children":2304},{},[2305,2310,2311,2317],{"type":44,"tag":117,"props":2306,"children":2307},{},[2308],{"type":49,"value":2309},"Ready check",{"type":49,"value":2265},{"type":44,"tag":58,"props":2312,"children":2314},{"className":2313},[],[2315],{"type":49,"value":2316},"--ready-check-timeout-sec 60",{"type":49,"value":2318}," to wait for the endpoint before benchmarking.",{"type":44,"tag":92,"props":2320,"children":2321},{},[2322,2327,2328,2334],{"type":44,"tag":117,"props":2323,"children":2324},{},[2325],{"type":49,"value":2326},"SSL",{"type":49,"value":2265},{"type":44,"tag":58,"props":2329,"children":2331},{"className":2330},[],[2332],{"type":49,"value":2333},"--insecure",{"type":49,"value":2335}," for self-signed certificates.",{"type":44,"tag":81,"props":2337,"children":2339},{"id":2338},"notes",[2340],{"type":49,"value":2341},"Notes",{"type":44,"tag":88,"props":2343,"children":2344},{},[2345,2370,2389],{"type":44,"tag":92,"props":2346,"children":2347},{},[2348,2350,2356,2357,2362,2364,2369],{"type":49,"value":2349},"For embeddings\u002Frerank benchmarks, use ",{"type":44,"tag":58,"props":2351,"children":2353},{"className":2352},[],[2354],{"type":49,"value":2355},"--backend openai-embeddings",{"type":49,"value":537},{"type":44,"tag":58,"props":2358,"children":2360},{"className":2359},[],[2361],{"type":49,"value":563},{"type":49,"value":2363},", or ",{"type":44,"tag":58,"props":2365,"children":2367},{"className":2366},[],[2368],{"type":49,"value":570},{"type":49,"value":2279},{"type":44,"tag":92,"props":2371,"children":2372},{},[2373,2379,2381,2387],{"type":44,"tag":58,"props":2374,"children":2376},{"className":2375},[],[2377],{"type":49,"value":2378},"--profile",{"type":49,"value":2380}," requires ",{"type":44,"tag":58,"props":2382,"children":2384},{"className":2383},[],[2385],{"type":49,"value":2386},"--profiler-config",{"type":49,"value":2388}," on the server for vLLM profiling.",{"type":44,"tag":92,"props":2390,"children":2391},{},[2392,2394,2401],{"type":49,"value":2393},"Goodput SLOs are useful for SLA-style analysis; see ",{"type":44,"tag":72,"props":2395,"children":2398},{"href":2396,"rel":2397},"https:\u002F\u002Farxiv.org\u002Fpdf\u002F2401.09670",[76],[2399],{"type":49,"value":2400},"DistServe paper",{"type":49,"value":2402}," for details.",{"type":44,"tag":2404,"props":2405,"children":2406},"style",{},[2407],{"type":49,"value":2408},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2410,"total":360},[2411,2428,2439,2447,2461,2477,2488],{"slug":2412,"name":2412,"fn":2413,"description":2414,"org":2415,"tags":2416,"stars":2425,"repoUrl":2426,"updatedAt":2427},"openclaw-vsr-bridge","integrate vLLM Semantic Router","Install vLLM Semantic Router in agent-safe mode, import supported OpenClaw model providers into canonical VSR config, and rewrite OpenClaw to target VSR.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2417,2420,2421,2424],{"name":2418,"slug":2419,"type":16},"Agents","agents",{"name":21,"slug":22,"type":16},{"name":2422,"slug":2423,"type":16},"Routing","routing",{"name":9,"slug":8,"type":16},4962,"https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fsemantic-router","2026-07-18T05:48:09.091161",{"slug":2429,"name":2429,"fn":2430,"description":2431,"org":2432,"tags":2433,"stars":27,"repoUrl":28,"updatedAt":2438},"vllm-bench-random-synthetic","run vLLM synthetic performance benchmarks","Run vLLM performance benchmark using synthetic random data to measure throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and other key performance metrics. Use when the user wants to quickly test vLLM serving performance without downloading external datasets.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2434,2435,2436,2437],{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:46:59.036927",{"slug":4,"name":4,"fn":5,"description":6,"org":2440,"tags":2441,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2442,2443,2444,2445,2446],{"name":25,"slug":26,"type":16},{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2448,"name":2448,"fn":2449,"description":2450,"org":2451,"tags":2452,"stars":27,"repoUrl":28,"updatedAt":2460},"vllm-deploy-docker","deploy vLLM using Docker containers","Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2453,2456,2458,2459],{"name":2454,"slug":2455,"type":16},"Deployment","deployment",{"name":2457,"slug":2067,"type":16},"Docker",{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:01.814419",{"slug":2462,"name":2462,"fn":2463,"description":2464,"org":2465,"tags":2466,"stars":27,"repoUrl":28,"updatedAt":2476},"vllm-deploy-k8s","deploy vLLM to Kubernetes clusters","Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint. Use this skill whenever the user wants to deploy, run, or serve vLLM on a Kubernetes cluster, including creating deployments, services, checking existing deployments, or managing vLLM on K8s.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2467,2470,2471,2474,2475],{"name":2468,"slug":2469,"type":16},"AI Infrastructure","ai-infrastructure",{"name":2454,"slug":2455,"type":16},{"name":2472,"slug":2473,"type":16},"Kubernetes","kubernetes",{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:46:58.556637",{"slug":2478,"name":2478,"fn":2479,"description":2480,"org":2481,"tags":2482,"stars":27,"repoUrl":28,"updatedAt":2487},"vllm-deploy-simple","deploy and test vLLM servers","Quick install and deploy vLLM, start serving with a simple LLM, and test OpenAI API.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2483,2484,2485,2486],{"name":25,"slug":26,"type":16},{"name":2454,"slug":2455,"type":16},{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:47:01.359655",{"slug":2489,"name":2489,"fn":2490,"description":2491,"org":2492,"tags":2493,"stars":27,"repoUrl":28,"updatedAt":2498},"vllm-prefix-cache-bench","benchmark vLLM prefix caching efficiency","This is a skill for benchmarking the efficiency of automatic prefix caching in vLLM using fixed prompts, real-world datasets, or synthetic prefix\u002Fsuffix patterns. Use when the user asks to benchmark prefix caching hit rate, caching efficiency, or repeated-prompt performance in vLLM.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2494,2495,2496,2497],{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},"2026-07-18T05:48:07.243092",{"items":2500,"total":236},[2501,2508,2516,2523,2531,2538],{"slug":2429,"name":2429,"fn":2430,"description":2431,"org":2502,"tags":2503,"stars":27,"repoUrl":28,"updatedAt":2438},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2504,2505,2506,2507],{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":2509,"tags":2510,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2511,2512,2513,2514,2515],{"name":25,"slug":26,"type":16},{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16},{"slug":2448,"name":2448,"fn":2449,"description":2450,"org":2517,"tags":2518,"stars":27,"repoUrl":28,"updatedAt":2460},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2519,2520,2521,2522],{"name":2454,"slug":2455,"type":16},{"name":2457,"slug":2067,"type":16},{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"slug":2462,"name":2462,"fn":2463,"description":2464,"org":2524,"tags":2525,"stars":27,"repoUrl":28,"updatedAt":2476},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2526,2527,2528,2529,2530],{"name":2468,"slug":2469,"type":16},{"name":2454,"slug":2455,"type":16},{"name":2472,"slug":2473,"type":16},{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"slug":2478,"name":2478,"fn":2479,"description":2480,"org":2532,"tags":2533,"stars":27,"repoUrl":28,"updatedAt":2487},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2534,2535,2536,2537],{"name":25,"slug":26,"type":16},{"name":2454,"slug":2455,"type":16},{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"slug":2489,"name":2489,"fn":2490,"description":2491,"org":2539,"tags":2540,"stars":27,"repoUrl":28,"updatedAt":2498},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2541,2542,2543,2544],{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":9,"slug":8,"type":16}]