[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-promptfoo-promptfoo-redteam-run":3,"mdc--1ne1i5-key":53,"related-repo-promptfoo-promptfoo-redteam-run":1506,"related-org-promptfoo-promptfoo-redteam-run":1552},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":10,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":48,"sourceUrl":51,"mdContent":52},"promptfoo-redteam-run","run promptfoo redteam security scans","Run, rerun, inspect, and QA promptfoo redteam scans from generated redteam YAML or an existing redteam setup config. Use when executing `promptfoo redteam eval` or `promptfoo redteam run`, exporting results, triaging attack success rate, grader failures, target errors, filter\u002Frerun commands, reports, or CI gates. Do not use for initial provider wiring or for choosing plugins and strategies before generation.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},"promptfoo","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fpromptfoo.png",[11,15,16,19,22],{"name":12,"slug":13,"type":14},"Security","security","tag",{"name":8,"slug":8,"type":14},{"name":17,"slug":18,"type":14},"Evals","evals",{"name":20,"slug":21,"type":14},"Code Analysis","code-analysis",{"name":23,"slug":24,"type":14},"Testing","testing",23182,"https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo","2026-07-18T05:48:03.576002",null,2079,[31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,24,47],"ci","ci-cd","cicd","evaluation","evaluation-framework","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","vulnerability-scanners",{"repoUrl":26,"stars":25,"forks":29,"topics":49,"description":50},[31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,24,47],"Test your prompts, agents, and RAGs. Red teaming\u002Fpentesting\u002Fvulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI\u002FCD integration.  Used by OpenAI and Anthropic.","https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo\u002Ftree\u002FHEAD\u002Fplugins\u002Fpromptfoo\u002Fskills\u002Fpromptfoo-redteam-run","---\nname: promptfoo-redteam-run\ndescription: >\n  Run, rerun, inspect, and QA promptfoo redteam scans from generated redteam YAML\n  or an existing redteam setup config. Use when executing `promptfoo redteam\n  eval` or `promptfoo redteam run`, exporting results, triaging attack success\n  rate, grader failures, target errors, filter\u002Frerun commands, reports, or CI\n  gates. Do not use for initial provider wiring or for choosing plugins and\n  strategies before generation.\n---\n\n# Promptfoo Redteam Run\n\nExecute redteam probes reproducibly, inspect the output artifact, and rerun only\nthe slice that needs attention. Prefer evaluating existing generated tests with\n`redteam eval`; regenerate with `redteam run` only when config\u002Ftest generation\nmust change.\n\nRead `references\u002Fredteam-run-patterns.md` when you need command recipes, result\ninspection snippets, or CI examples.\n\n## Inputs\n\nInfer these from the repo or user prompt:\n\n- Generated scan file, usually `redteam.yaml`, or setup config if regeneration\n  is requested.\n- Target environment, secrets\u002Fenv file, concurrency\u002Frate limits, and whether\n  cloud sharing is allowed.\n- Grading mode: default remote grading, `redteam.provider`, `--grader`, or local\n  deterministic QA provider.\n- Desired output: JSON\u002FYAML\u002FHTML export, report, CI gate, failure triage, or\n  rerun of a previous result.\n\nIf a target or generated tests are missing, use `promptfoo-provider-setup` or\n`promptfoo-redteam-setup` first.\n\n## Workflow\n\n### 1. Choose run mode\n\n- Use `redteam eval` when `redteam.yaml` already exists and you want stable\n  apples-to-apples runs.\n- Use `redteam run --force` only when the setup changed or the user wants fresh\n  generated probes.\n- Use a configured `redteam.provider` or `--grader` only when the scan needs a\n  specific generator\u002Fgrader or deterministic QA behavior.\n- Disable cloud sharing by default for internal targets. `redteam eval` and\n  `retry` accept `--no-share`; `redteam run` does not currently expose that\n  flag, so export `PROMPTFOO_DISABLE_SHARING=true` for the whole invocation or\n  split into `redteam generate` + `redteam eval --no-share`. Only re-enable\n  sharing when the user explicitly asks for a cloud URL.\n\n### 2. Preflight\n\nUse the CLI form that matches your environment for every command below: from the\npromptfoo repo, `npm run local -- redteam …` (align Node first with\n`source ~\u002F.nvm\u002Fnvm.sh && nvm use`); outside the repo (an installed plugin or your\nown app project), `npx promptfoo@latest redteam …`, or a globally installed\n`promptfoo redteam …`.\n\nValidate first:\n\n```bash\nnpm run local -- validate config -c path\u002Fto\u002Fredteam.yaml\nnpm run local -- validate target -c path\u002Fto\u002Fredteam.yaml\n# Outside the repo:\nnpx promptfoo@latest validate config -c path\u002Fto\u002Fredteam.yaml\n```\n\nCheck that generated tests include `assert`, `metadata.pluginId`,\n`metadata.purpose` or `defaultTest.metadata.purpose`, and the real input vars.\nFor `file:\u002F\u002F` providers in a generated eval file, target providers resolve like\nnormal config file providers, so `file:\u002F\u002F.\u002Ftarget.mjs` is relative to that config\nfile. `redteam.provider` is loaded during grading from the command working\ndirectory, so use an absolute path or a repo-root-relative path when running from\nthe repo root.\n\nIf validation fails with `ENOENT` for `file:\u002F\u002F.\u002Ftarget.mjs`, the generated YAML\nwas probably written to a different directory than the target. Regenerate beside\nthe source config, move the generated file next to the target, or change the\ntarget id to an absolute\u002Frepo-root-relative `file:\u002F\u002F` path before rerunning.\n\n### 3. Run and export\n\nEvaluate generated tests:\n\n```bash\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml -o \u002Ftmp\u002Fredteam-results.json --no-cache --no-share --no-progress-bar\n# Outside the repo:\nnpx promptfoo@latest redteam eval -c path\u002Fto\u002Fredteam.yaml -o \u002Ftmp\u002Fredteam-results.json --no-cache --no-share --no-progress-bar\n```\n\nGenerate and evaluate in one command only when needed. `redteam run` has no\n`--no-share` flag, so disable sharing via the environment variable:\n\n```bash\nPROMPTFOO_DISABLE_SHARING=true npm run local -- redteam run -c path\u002Fto\u002Fpromptfooconfig.yaml --force --no-cache --no-progress-bar\n```\n\nFor fragile targets, set `-j 1` and add `--delay` rather than allowing broad\nconcurrency.\n\n### 4. Inspect results\n\nAlways inspect the exported artifact. Do not rely only on the exit code because\nredteam failures may intentionally return a failing exit status.\n\nLook for:\n\n- `results.stats.successes`, `failures`, `errors`, and `tokenUsage`\n- Failed or errored rows, including `response.output`, `error`, `gradingResult`,\n  `metadata.pluginId`, `metadata.strategyId`, and target label\n- `shareableUrl`; it should be `null` when `--no-share` is used\n- Attack success rate: `failures \u002F (successes + failures)`\n\nTreat grader transport\u002Fparse failures separately from real target failures.\nIf `--filter-errors-only` returns zero rows, the source result likely had no\nERROR rows or the generated test indices changed since the source run.\n\n### 5. Rerun narrowly\n\nUse filters before rerunning expensive scans:\n\n```bash\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-failing \u002Ftmp\u002Fredteam-results.json -o \u002Ftmp\u002Fredteam-failing-rerun.json --no-cache --no-share --no-progress-bar\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-errors-only \u002Ftmp\u002Fredteam-results.json -o \u002Ftmp\u002Fredteam-errors-rerun.json --no-cache --no-share --no-progress-bar\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-metadata pluginId=policy -o \u002Ftmp\u002Fredteam-policy.json --no-cache --no-share --no-progress-bar\n```\n\nFor error-only reruns that should update the original evaluation in place, use\n`promptfoo retry \u003CevalId>` instead of creating another eval.\n\n### 6. Report or gate\n\nUse `redteam report` for interactive triage after results are written. It starts\nor reuses the local Promptfoo UI, so ask before running it unless the user\nexplicitly requested the report UI:\n\n```bash\nnpm run local -- redteam report\n```\n\nFor CI, gate on explicit metrics from the JSON export. Keep thresholds tied to\nthe app's risk tolerance and track category-level changes with\n`metadata.pluginId`.\n\n## Common Mistakes\n\n```bash\n# WRONG: regenerates probes when you only wanted a comparable rerun\npromptfoo redteam run\n\n# BETTER: reuse generated tests\npromptfoo redteam eval -c redteam.yaml -o results.json --no-cache --no-share\n```\n\n```bash\n# WRONG: broad rerun after a flaky target error\npromptfoo redteam eval -c redteam.yaml\n\n# BETTER: rerun only target\u002Fgrader errors from the prior result\npromptfoo redteam eval -c redteam.yaml --filter-errors-only results.json -o errors-rerun.json --no-cache --no-share\n```\n\n## Output Contract\n\nWhen done, state:\n\n- Run mode used: existing generated tests or regenerate-and-run\n- Config\u002Fresult\u002Freport paths\n- Data-sharing mode and grader\u002Fprovider used\n- Commands run and whether any returned nonzero due to redteam failures\n- Success, failure, error counts and attack success rate\n- Highest-priority failing plugins\u002Fstrategies and recommended next rerun or fix\n",{"data":54,"body":55},{"name":4,"description":6},{"type":56,"children":57},"root",[58,66,89,102,109,114,163,184,190,197,309,315,352,357,493,551,578,584,589,713,732,806,827,833,838,843,953,966,972,977,1180,1193,1199,1211,1246,1257,1263,1354,1456,1462,1467,1500],{"type":59,"tag":60,"props":61,"children":62},"element","h1",{"id":4},[63],{"type":64,"value":65},"text","Promptfoo Redteam Run",{"type":59,"tag":67,"props":68,"children":69},"p",{},[70,72,79,81,87],{"type":64,"value":71},"Execute redteam probes reproducibly, inspect the output artifact, and rerun only\nthe slice that needs attention. Prefer evaluating existing generated tests with\n",{"type":59,"tag":73,"props":74,"children":76},"code",{"className":75},[],[77],{"type":64,"value":78},"redteam eval",{"type":64,"value":80},"; regenerate with ",{"type":59,"tag":73,"props":82,"children":84},{"className":83},[],[85],{"type":64,"value":86},"redteam run",{"type":64,"value":88}," only when config\u002Ftest generation\nmust change.",{"type":59,"tag":67,"props":90,"children":91},{},[92,94,100],{"type":64,"value":93},"Read ",{"type":59,"tag":73,"props":95,"children":97},{"className":96},[],[98],{"type":64,"value":99},"references\u002Fredteam-run-patterns.md",{"type":64,"value":101}," when you need command recipes, result\ninspection snippets, or CI examples.",{"type":59,"tag":103,"props":104,"children":106},"h2",{"id":105},"inputs",[107],{"type":64,"value":108},"Inputs",{"type":59,"tag":67,"props":110,"children":111},{},[112],{"type":64,"value":113},"Infer these from the repo or user prompt:",{"type":59,"tag":115,"props":116,"children":117},"ul",{},[118,132,137,158],{"type":59,"tag":119,"props":120,"children":121},"li",{},[122,124,130],{"type":64,"value":123},"Generated scan file, usually ",{"type":59,"tag":73,"props":125,"children":127},{"className":126},[],[128],{"type":64,"value":129},"redteam.yaml",{"type":64,"value":131},", or setup config if regeneration\nis requested.",{"type":59,"tag":119,"props":133,"children":134},{},[135],{"type":64,"value":136},"Target environment, secrets\u002Fenv file, concurrency\u002Frate limits, and whether\ncloud sharing is allowed.",{"type":59,"tag":119,"props":138,"children":139},{},[140,142,148,150,156],{"type":64,"value":141},"Grading mode: default remote grading, ",{"type":59,"tag":73,"props":143,"children":145},{"className":144},[],[146],{"type":64,"value":147},"redteam.provider",{"type":64,"value":149},", ",{"type":59,"tag":73,"props":151,"children":153},{"className":152},[],[154],{"type":64,"value":155},"--grader",{"type":64,"value":157},", or local\ndeterministic QA provider.",{"type":59,"tag":119,"props":159,"children":160},{},[161],{"type":64,"value":162},"Desired output: JSON\u002FYAML\u002FHTML export, report, CI gate, failure triage, or\nrerun of a previous result.",{"type":59,"tag":67,"props":164,"children":165},{},[166,168,174,176,182],{"type":64,"value":167},"If a target or generated tests are missing, use ",{"type":59,"tag":73,"props":169,"children":171},{"className":170},[],[172],{"type":64,"value":173},"promptfoo-provider-setup",{"type":64,"value":175}," or\n",{"type":59,"tag":73,"props":177,"children":179},{"className":178},[],[180],{"type":64,"value":181},"promptfoo-redteam-setup",{"type":64,"value":183}," first.",{"type":59,"tag":103,"props":185,"children":187},{"id":186},"workflow",[188],{"type":64,"value":189},"Workflow",{"type":59,"tag":191,"props":192,"children":194},"h3",{"id":193},"_1-choose-run-mode",[195],{"type":64,"value":196},"1. Choose run mode",{"type":59,"tag":115,"props":198,"children":199},{},[200,219,231,250],{"type":59,"tag":119,"props":201,"children":202},{},[203,205,210,212,217],{"type":64,"value":204},"Use ",{"type":59,"tag":73,"props":206,"children":208},{"className":207},[],[209],{"type":64,"value":78},{"type":64,"value":211}," when ",{"type":59,"tag":73,"props":213,"children":215},{"className":214},[],[216],{"type":64,"value":129},{"type":64,"value":218}," already exists and you want stable\napples-to-apples runs.",{"type":59,"tag":119,"props":220,"children":221},{},[222,223,229],{"type":64,"value":204},{"type":59,"tag":73,"props":224,"children":226},{"className":225},[],[227],{"type":64,"value":228},"redteam run --force",{"type":64,"value":230}," only when the setup changed or the user wants fresh\ngenerated probes.",{"type":59,"tag":119,"props":232,"children":233},{},[234,236,241,243,248],{"type":64,"value":235},"Use a configured ",{"type":59,"tag":73,"props":237,"children":239},{"className":238},[],[240],{"type":64,"value":147},{"type":64,"value":242}," or ",{"type":59,"tag":73,"props":244,"children":246},{"className":245},[],[247],{"type":64,"value":155},{"type":64,"value":249}," only when the scan needs a\nspecific generator\u002Fgrader or deterministic QA behavior.",{"type":59,"tag":119,"props":251,"children":252},{},[253,255,260,262,268,270,276,278,283,285,291,293,299,301,307],{"type":64,"value":254},"Disable cloud sharing by default for internal targets. ",{"type":59,"tag":73,"props":256,"children":258},{"className":257},[],[259],{"type":64,"value":78},{"type":64,"value":261}," and\n",{"type":59,"tag":73,"props":263,"children":265},{"className":264},[],[266],{"type":64,"value":267},"retry",{"type":64,"value":269}," accept ",{"type":59,"tag":73,"props":271,"children":273},{"className":272},[],[274],{"type":64,"value":275},"--no-share",{"type":64,"value":277},"; ",{"type":59,"tag":73,"props":279,"children":281},{"className":280},[],[282],{"type":64,"value":86},{"type":64,"value":284}," does not currently expose that\nflag, so export ",{"type":59,"tag":73,"props":286,"children":288},{"className":287},[],[289],{"type":64,"value":290},"PROMPTFOO_DISABLE_SHARING=true",{"type":64,"value":292}," for the whole invocation or\nsplit into ",{"type":59,"tag":73,"props":294,"children":296},{"className":295},[],[297],{"type":64,"value":298},"redteam generate",{"type":64,"value":300}," + ",{"type":59,"tag":73,"props":302,"children":304},{"className":303},[],[305],{"type":64,"value":306},"redteam eval --no-share",{"type":64,"value":308},". Only re-enable\nsharing when the user explicitly asks for a cloud URL.",{"type":59,"tag":191,"props":310,"children":312},{"id":311},"_2-preflight",[313],{"type":64,"value":314},"2. Preflight",{"type":59,"tag":67,"props":316,"children":317},{},[318,320,326,328,334,336,342,344,350],{"type":64,"value":319},"Use the CLI form that matches your environment for every command below: from the\npromptfoo repo, ",{"type":59,"tag":73,"props":321,"children":323},{"className":322},[],[324],{"type":64,"value":325},"npm run local -- redteam …",{"type":64,"value":327}," (align Node first with\n",{"type":59,"tag":73,"props":329,"children":331},{"className":330},[],[332],{"type":64,"value":333},"source ~\u002F.nvm\u002Fnvm.sh && nvm use",{"type":64,"value":335},"); outside the repo (an installed plugin or your\nown app project), ",{"type":59,"tag":73,"props":337,"children":339},{"className":338},[],[340],{"type":64,"value":341},"npx promptfoo@latest redteam …",{"type":64,"value":343},", or a globally installed\n",{"type":59,"tag":73,"props":345,"children":347},{"className":346},[],[348],{"type":64,"value":349},"promptfoo redteam …",{"type":64,"value":351},".",{"type":59,"tag":67,"props":353,"children":354},{},[355],{"type":64,"value":356},"Validate first:",{"type":59,"tag":358,"props":359,"children":364},"pre",{"className":360,"code":361,"language":362,"meta":363,"style":363},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","npm run local -- validate config -c path\u002Fto\u002Fredteam.yaml\nnpm run local -- validate target -c path\u002Fto\u002Fredteam.yaml\n# Outside the repo:\nnpx promptfoo@latest validate config -c path\u002Fto\u002Fredteam.yaml\n","bash","",[365],{"type":59,"tag":73,"props":366,"children":367},{"__ignoreMap":363},[368,416,453,463],{"type":59,"tag":369,"props":370,"children":373},"span",{"class":371,"line":372},"line",1,[374,380,386,391,396,401,406,411],{"type":59,"tag":369,"props":375,"children":377},{"style":376},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[378],{"type":64,"value":379},"npm",{"type":59,"tag":369,"props":381,"children":383},{"style":382},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[384],{"type":64,"value":385}," run",{"type":59,"tag":369,"props":387,"children":388},{"style":382},[389],{"type":64,"value":390}," local",{"type":59,"tag":369,"props":392,"children":393},{"style":382},[394],{"type":64,"value":395}," --",{"type":59,"tag":369,"props":397,"children":398},{"style":382},[399],{"type":64,"value":400}," validate",{"type":59,"tag":369,"props":402,"children":403},{"style":382},[404],{"type":64,"value":405}," config",{"type":59,"tag":369,"props":407,"children":408},{"style":382},[409],{"type":64,"value":410}," -c",{"type":59,"tag":369,"props":412,"children":413},{"style":382},[414],{"type":64,"value":415}," path\u002Fto\u002Fredteam.yaml\n",{"type":59,"tag":369,"props":417,"children":419},{"class":371,"line":418},2,[420,424,428,432,436,440,445,449],{"type":59,"tag":369,"props":421,"children":422},{"style":376},[423],{"type":64,"value":379},{"type":59,"tag":369,"props":425,"children":426},{"style":382},[427],{"type":64,"value":385},{"type":59,"tag":369,"props":429,"children":430},{"style":382},[431],{"type":64,"value":390},{"type":59,"tag":369,"props":433,"children":434},{"style":382},[435],{"type":64,"value":395},{"type":59,"tag":369,"props":437,"children":438},{"style":382},[439],{"type":64,"value":400},{"type":59,"tag":369,"props":441,"children":442},{"style":382},[443],{"type":64,"value":444}," target",{"type":59,"tag":369,"props":446,"children":447},{"style":382},[448],{"type":64,"value":410},{"type":59,"tag":369,"props":450,"children":451},{"style":382},[452],{"type":64,"value":415},{"type":59,"tag":369,"props":454,"children":456},{"class":371,"line":455},3,[457],{"type":59,"tag":369,"props":458,"children":460},{"style":459},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[461],{"type":64,"value":462},"# Outside the repo:\n",{"type":59,"tag":369,"props":464,"children":466},{"class":371,"line":465},4,[467,472,477,481,485,489],{"type":59,"tag":369,"props":468,"children":469},{"style":376},[470],{"type":64,"value":471},"npx",{"type":59,"tag":369,"props":473,"children":474},{"style":382},[475],{"type":64,"value":476}," promptfoo@latest",{"type":59,"tag":369,"props":478,"children":479},{"style":382},[480],{"type":64,"value":400},{"type":59,"tag":369,"props":482,"children":483},{"style":382},[484],{"type":64,"value":405},{"type":59,"tag":369,"props":486,"children":487},{"style":382},[488],{"type":64,"value":410},{"type":59,"tag":369,"props":490,"children":491},{"style":382},[492],{"type":64,"value":415},{"type":59,"tag":67,"props":494,"children":495},{},[496,498,504,505,511,513,519,520,526,528,534,536,542,544,549],{"type":64,"value":497},"Check that generated tests include ",{"type":59,"tag":73,"props":499,"children":501},{"className":500},[],[502],{"type":64,"value":503},"assert",{"type":64,"value":149},{"type":59,"tag":73,"props":506,"children":508},{"className":507},[],[509],{"type":64,"value":510},"metadata.pluginId",{"type":64,"value":512},",\n",{"type":59,"tag":73,"props":514,"children":516},{"className":515},[],[517],{"type":64,"value":518},"metadata.purpose",{"type":64,"value":242},{"type":59,"tag":73,"props":521,"children":523},{"className":522},[],[524],{"type":64,"value":525},"defaultTest.metadata.purpose",{"type":64,"value":527},", and the real input vars.\nFor ",{"type":59,"tag":73,"props":529,"children":531},{"className":530},[],[532],{"type":64,"value":533},"file:\u002F\u002F",{"type":64,"value":535}," providers in a generated eval file, target providers resolve like\nnormal config file providers, so ",{"type":59,"tag":73,"props":537,"children":539},{"className":538},[],[540],{"type":64,"value":541},"file:\u002F\u002F.\u002Ftarget.mjs",{"type":64,"value":543}," is relative to that config\nfile. ",{"type":59,"tag":73,"props":545,"children":547},{"className":546},[],[548],{"type":64,"value":147},{"type":64,"value":550}," is loaded during grading from the command working\ndirectory, so use an absolute path or a repo-root-relative path when running from\nthe repo root.",{"type":59,"tag":67,"props":552,"children":553},{},[554,556,562,564,569,571,576],{"type":64,"value":555},"If validation fails with ",{"type":59,"tag":73,"props":557,"children":559},{"className":558},[],[560],{"type":64,"value":561},"ENOENT",{"type":64,"value":563}," for ",{"type":59,"tag":73,"props":565,"children":567},{"className":566},[],[568],{"type":64,"value":541},{"type":64,"value":570},", the generated YAML\nwas probably written to a different directory than the target. Regenerate beside\nthe source config, move the generated file next to the target, or change the\ntarget id to an absolute\u002Frepo-root-relative ",{"type":59,"tag":73,"props":572,"children":574},{"className":573},[],[575],{"type":64,"value":533},{"type":64,"value":577}," path before rerunning.",{"type":59,"tag":191,"props":579,"children":581},{"id":580},"_3-run-and-export",[582],{"type":64,"value":583},"3. Run and export",{"type":59,"tag":67,"props":585,"children":586},{},[587],{"type":64,"value":588},"Evaluate generated tests:",{"type":59,"tag":358,"props":590,"children":592},{"className":360,"code":591,"language":362,"meta":363,"style":363},"npm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml -o \u002Ftmp\u002Fredteam-results.json --no-cache --no-share --no-progress-bar\n# Outside the repo:\nnpx promptfoo@latest redteam eval -c path\u002Fto\u002Fredteam.yaml -o \u002Ftmp\u002Fredteam-results.json --no-cache --no-share --no-progress-bar\n",[593],{"type":59,"tag":73,"props":594,"children":595},{"__ignoreMap":363},[596,659,666],{"type":59,"tag":369,"props":597,"children":598},{"class":371,"line":372},[599,603,607,611,615,620,625,629,634,639,644,649,654],{"type":59,"tag":369,"props":600,"children":601},{"style":376},[602],{"type":64,"value":379},{"type":59,"tag":369,"props":604,"children":605},{"style":382},[606],{"type":64,"value":385},{"type":59,"tag":369,"props":608,"children":609},{"style":382},[610],{"type":64,"value":390},{"type":59,"tag":369,"props":612,"children":613},{"style":382},[614],{"type":64,"value":395},{"type":59,"tag":369,"props":616,"children":617},{"style":382},[618],{"type":64,"value":619}," redteam",{"type":59,"tag":369,"props":621,"children":622},{"style":382},[623],{"type":64,"value":624}," eval",{"type":59,"tag":369,"props":626,"children":627},{"style":382},[628],{"type":64,"value":410},{"type":59,"tag":369,"props":630,"children":631},{"style":382},[632],{"type":64,"value":633}," path\u002Fto\u002Fredteam.yaml",{"type":59,"tag":369,"props":635,"children":636},{"style":382},[637],{"type":64,"value":638}," -o",{"type":59,"tag":369,"props":640,"children":641},{"style":382},[642],{"type":64,"value":643}," \u002Ftmp\u002Fredteam-results.json",{"type":59,"tag":369,"props":645,"children":646},{"style":382},[647],{"type":64,"value":648}," --no-cache",{"type":59,"tag":369,"props":650,"children":651},{"style":382},[652],{"type":64,"value":653}," --no-share",{"type":59,"tag":369,"props":655,"children":656},{"style":382},[657],{"type":64,"value":658}," --no-progress-bar\n",{"type":59,"tag":369,"props":660,"children":661},{"class":371,"line":418},[662],{"type":59,"tag":369,"props":663,"children":664},{"style":459},[665],{"type":64,"value":462},{"type":59,"tag":369,"props":667,"children":668},{"class":371,"line":455},[669,673,677,681,685,689,693,697,701,705,709],{"type":59,"tag":369,"props":670,"children":671},{"style":376},[672],{"type":64,"value":471},{"type":59,"tag":369,"props":674,"children":675},{"style":382},[676],{"type":64,"value":476},{"type":59,"tag":369,"props":678,"children":679},{"style":382},[680],{"type":64,"value":619},{"type":59,"tag":369,"props":682,"children":683},{"style":382},[684],{"type":64,"value":624},{"type":59,"tag":369,"props":686,"children":687},{"style":382},[688],{"type":64,"value":410},{"type":59,"tag":369,"props":690,"children":691},{"style":382},[692],{"type":64,"value":633},{"type":59,"tag":369,"props":694,"children":695},{"style":382},[696],{"type":64,"value":638},{"type":59,"tag":369,"props":698,"children":699},{"style":382},[700],{"type":64,"value":643},{"type":59,"tag":369,"props":702,"children":703},{"style":382},[704],{"type":64,"value":648},{"type":59,"tag":369,"props":706,"children":707},{"style":382},[708],{"type":64,"value":653},{"type":59,"tag":369,"props":710,"children":711},{"style":382},[712],{"type":64,"value":658},{"type":59,"tag":67,"props":714,"children":715},{},[716,718,723,725,730],{"type":64,"value":717},"Generate and evaluate in one command only when needed. ",{"type":59,"tag":73,"props":719,"children":721},{"className":720},[],[722],{"type":64,"value":86},{"type":64,"value":724}," has no\n",{"type":59,"tag":73,"props":726,"children":728},{"className":727},[],[729],{"type":64,"value":275},{"type":64,"value":731}," flag, so disable sharing via the environment variable:",{"type":59,"tag":358,"props":733,"children":735},{"className":360,"code":734,"language":362,"meta":363,"style":363},"PROMPTFOO_DISABLE_SHARING=true npm run local -- redteam run -c path\u002Fto\u002Fpromptfooconfig.yaml --force --no-cache --no-progress-bar\n",[736],{"type":59,"tag":73,"props":737,"children":738},{"__ignoreMap":363},[739],{"type":59,"tag":369,"props":740,"children":741},{"class":371,"line":372},[742,748,754,759,764,768,772,776,780,784,788,793,798,802],{"type":59,"tag":369,"props":743,"children":745},{"style":744},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[746],{"type":64,"value":747},"PROMPTFOO_DISABLE_SHARING",{"type":59,"tag":369,"props":749,"children":751},{"style":750},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[752],{"type":64,"value":753},"=",{"type":59,"tag":369,"props":755,"children":756},{"style":382},[757],{"type":64,"value":758},"true",{"type":59,"tag":369,"props":760,"children":761},{"style":376},[762],{"type":64,"value":763}," npm",{"type":59,"tag":369,"props":765,"children":766},{"style":382},[767],{"type":64,"value":385},{"type":59,"tag":369,"props":769,"children":770},{"style":382},[771],{"type":64,"value":390},{"type":59,"tag":369,"props":773,"children":774},{"style":382},[775],{"type":64,"value":395},{"type":59,"tag":369,"props":777,"children":778},{"style":382},[779],{"type":64,"value":619},{"type":59,"tag":369,"props":781,"children":782},{"style":382},[783],{"type":64,"value":385},{"type":59,"tag":369,"props":785,"children":786},{"style":382},[787],{"type":64,"value":410},{"type":59,"tag":369,"props":789,"children":790},{"style":382},[791],{"type":64,"value":792}," path\u002Fto\u002Fpromptfooconfig.yaml",{"type":59,"tag":369,"props":794,"children":795},{"style":382},[796],{"type":64,"value":797}," --force",{"type":59,"tag":369,"props":799,"children":800},{"style":382},[801],{"type":64,"value":648},{"type":59,"tag":369,"props":803,"children":804},{"style":382},[805],{"type":64,"value":658},{"type":59,"tag":67,"props":807,"children":808},{},[809,811,817,819,825],{"type":64,"value":810},"For fragile targets, set ",{"type":59,"tag":73,"props":812,"children":814},{"className":813},[],[815],{"type":64,"value":816},"-j 1",{"type":64,"value":818}," and add ",{"type":59,"tag":73,"props":820,"children":822},{"className":821},[],[823],{"type":64,"value":824},"--delay",{"type":64,"value":826}," rather than allowing broad\nconcurrency.",{"type":59,"tag":191,"props":828,"children":830},{"id":829},"_4-inspect-results",[831],{"type":64,"value":832},"4. Inspect results",{"type":59,"tag":67,"props":834,"children":835},{},[836],{"type":64,"value":837},"Always inspect the exported artifact. Do not rely only on the exit code because\nredteam failures may intentionally return a failing exit status.",{"type":59,"tag":67,"props":839,"children":840},{},[841],{"type":64,"value":842},"Look for:",{"type":59,"tag":115,"props":844,"children":845},{},[846,877,917,942],{"type":59,"tag":119,"props":847,"children":848},{},[849,855,856,862,863,869,871],{"type":59,"tag":73,"props":850,"children":852},{"className":851},[],[853],{"type":64,"value":854},"results.stats.successes",{"type":64,"value":149},{"type":59,"tag":73,"props":857,"children":859},{"className":858},[],[860],{"type":64,"value":861},"failures",{"type":64,"value":149},{"type":59,"tag":73,"props":864,"children":866},{"className":865},[],[867],{"type":64,"value":868},"errors",{"type":64,"value":870},", and ",{"type":59,"tag":73,"props":872,"children":874},{"className":873},[],[875],{"type":64,"value":876},"tokenUsage",{"type":59,"tag":119,"props":878,"children":879},{},[880,882,888,889,895,896,902,903,908,909,915],{"type":64,"value":881},"Failed or errored rows, including ",{"type":59,"tag":73,"props":883,"children":885},{"className":884},[],[886],{"type":64,"value":887},"response.output",{"type":64,"value":149},{"type":59,"tag":73,"props":890,"children":892},{"className":891},[],[893],{"type":64,"value":894},"error",{"type":64,"value":149},{"type":59,"tag":73,"props":897,"children":899},{"className":898},[],[900],{"type":64,"value":901},"gradingResult",{"type":64,"value":512},{"type":59,"tag":73,"props":904,"children":906},{"className":905},[],[907],{"type":64,"value":510},{"type":64,"value":149},{"type":59,"tag":73,"props":910,"children":912},{"className":911},[],[913],{"type":64,"value":914},"metadata.strategyId",{"type":64,"value":916},", and target label",{"type":59,"tag":119,"props":918,"children":919},{},[920,926,928,934,935,940],{"type":59,"tag":73,"props":921,"children":923},{"className":922},[],[924],{"type":64,"value":925},"shareableUrl",{"type":64,"value":927},"; it should be ",{"type":59,"tag":73,"props":929,"children":931},{"className":930},[],[932],{"type":64,"value":933},"null",{"type":64,"value":211},{"type":59,"tag":73,"props":936,"children":938},{"className":937},[],[939],{"type":64,"value":275},{"type":64,"value":941}," is used",{"type":59,"tag":119,"props":943,"children":944},{},[945,947],{"type":64,"value":946},"Attack success rate: ",{"type":59,"tag":73,"props":948,"children":950},{"className":949},[],[951],{"type":64,"value":952},"failures \u002F (successes + failures)",{"type":59,"tag":67,"props":954,"children":955},{},[956,958,964],{"type":64,"value":957},"Treat grader transport\u002Fparse failures separately from real target failures.\nIf ",{"type":59,"tag":73,"props":959,"children":961},{"className":960},[],[962],{"type":64,"value":963},"--filter-errors-only",{"type":64,"value":965}," returns zero rows, the source result likely had no\nERROR rows or the generated test indices changed since the source run.",{"type":59,"tag":191,"props":967,"children":969},{"id":968},"_5-rerun-narrowly",[970],{"type":64,"value":971},"5. Rerun narrowly",{"type":59,"tag":67,"props":973,"children":974},{},[975],{"type":64,"value":976},"Use filters before rerunning expensive scans:",{"type":59,"tag":358,"props":978,"children":980},{"className":360,"code":979,"language":362,"meta":363,"style":363},"npm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-failing \u002Ftmp\u002Fredteam-results.json -o \u002Ftmp\u002Fredteam-failing-rerun.json --no-cache --no-share --no-progress-bar\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-errors-only \u002Ftmp\u002Fredteam-results.json -o \u002Ftmp\u002Fredteam-errors-rerun.json --no-cache --no-share --no-progress-bar\nnpm run local -- redteam eval -c path\u002Fto\u002Fredteam.yaml --filter-metadata pluginId=policy -o \u002Ftmp\u002Fredteam-policy.json --no-cache --no-share --no-progress-bar\n",[981],{"type":59,"tag":73,"props":982,"children":983},{"__ignoreMap":363},[984,1049,1114],{"type":59,"tag":369,"props":985,"children":986},{"class":371,"line":372},[987,991,995,999,1003,1007,1011,1015,1019,1024,1028,1032,1037,1041,1045],{"type":59,"tag":369,"props":988,"children":989},{"style":376},[990],{"type":64,"value":379},{"type":59,"tag":369,"props":992,"children":993},{"style":382},[994],{"type":64,"value":385},{"type":59,"tag":369,"props":996,"children":997},{"style":382},[998],{"type":64,"value":390},{"type":59,"tag":369,"props":1000,"children":1001},{"style":382},[1002],{"type":64,"value":395},{"type":59,"tag":369,"props":1004,"children":1005},{"style":382},[1006],{"type":64,"value":619},{"type":59,"tag":369,"props":1008,"children":1009},{"style":382},[1010],{"type":64,"value":624},{"type":59,"tag":369,"props":1012,"children":1013},{"style":382},[1014],{"type":64,"value":410},{"type":59,"tag":369,"props":1016,"children":1017},{"style":382},[1018],{"type":64,"value":633},{"type":59,"tag":369,"props":1020,"children":1021},{"style":382},[1022],{"type":64,"value":1023}," --filter-failing",{"type":59,"tag":369,"props":1025,"children":1026},{"style":382},[1027],{"type":64,"value":643},{"type":59,"tag":369,"props":1029,"children":1030},{"style":382},[1031],{"type":64,"value":638},{"type":59,"tag":369,"props":1033,"children":1034},{"style":382},[1035],{"type":64,"value":1036}," \u002Ftmp\u002Fredteam-failing-rerun.json",{"type":59,"tag":369,"props":1038,"children":1039},{"style":382},[1040],{"type":64,"value":648},{"type":59,"tag":369,"props":1042,"children":1043},{"style":382},[1044],{"type":64,"value":653},{"type":59,"tag":369,"props":1046,"children":1047},{"style":382},[1048],{"type":64,"value":658},{"type":59,"tag":369,"props":1050,"children":1051},{"class":371,"line":418},[1052,1056,1060,1064,1068,1072,1076,1080,1084,1089,1093,1097,1102,1106,1110],{"type":59,"tag":369,"props":1053,"children":1054},{"style":376},[1055],{"type":64,"value":379},{"type":59,"tag":369,"props":1057,"children":1058},{"style":382},[1059],{"type":64,"value":385},{"type":59,"tag":369,"props":1061,"children":1062},{"style":382},[1063],{"type":64,"value":390},{"type":59,"tag":369,"props":1065,"children":1066},{"style":382},[1067],{"type":64,"value":395},{"type":59,"tag":369,"props":1069,"children":1070},{"style":382},[1071],{"type":64,"value":619},{"type":59,"tag":369,"props":1073,"children":1074},{"style":382},[1075],{"type":64,"value":624},{"type":59,"tag":369,"props":1077,"children":1078},{"style":382},[1079],{"type":64,"value":410},{"type":59,"tag":369,"props":1081,"children":1082},{"style":382},[1083],{"type":64,"value":633},{"type":59,"tag":369,"props":1085,"children":1086},{"style":382},[1087],{"type":64,"value":1088}," --filter-errors-only",{"type":59,"tag":369,"props":1090,"children":1091},{"style":382},[1092],{"type":64,"value":643},{"type":59,"tag":369,"props":1094,"children":1095},{"style":382},[1096],{"type":64,"value":638},{"type":59,"tag":369,"props":1098,"children":1099},{"style":382},[1100],{"type":64,"value":1101}," \u002Ftmp\u002Fredteam-errors-rerun.json",{"type":59,"tag":369,"props":1103,"children":1104},{"style":382},[1105],{"type":64,"value":648},{"type":59,"tag":369,"props":1107,"children":1108},{"style":382},[1109],{"type":64,"value":653},{"type":59,"tag":369,"props":1111,"children":1112},{"style":382},[1113],{"type":64,"value":658},{"type":59,"tag":369,"props":1115,"children":1116},{"class":371,"line":455},[1117,1121,1125,1129,1133,1137,1141,1145,1149,1154,1159,1163,1168,1172,1176],{"type":59,"tag":369,"props":1118,"children":1119},{"style":376},[1120],{"type":64,"value":379},{"type":59,"tag":369,"props":1122,"children":1123},{"style":382},[1124],{"type":64,"value":385},{"type":59,"tag":369,"props":1126,"children":1127},{"style":382},[1128],{"type":64,"value":390},{"type":59,"tag":369,"props":1130,"children":1131},{"style":382},[1132],{"type":64,"value":395},{"type":59,"tag":369,"props":1134,"children":1135},{"style":382},[1136],{"type":64,"value":619},{"type":59,"tag":369,"props":1138,"children":1139},{"style":382},[1140],{"type":64,"value":624},{"type":59,"tag":369,"props":1142,"children":1143},{"style":382},[1144],{"type":64,"value":410},{"type":59,"tag":369,"props":1146,"children":1147},{"style":382},[1148],{"type":64,"value":633},{"type":59,"tag":369,"props":1150,"children":1151},{"style":382},[1152],{"type":64,"value":1153}," --filter-metadata",{"type":59,"tag":369,"props":1155,"children":1156},{"style":382},[1157],{"type":64,"value":1158}," pluginId=policy",{"type":59,"tag":369,"props":1160,"children":1161},{"style":382},[1162],{"type":64,"value":638},{"type":59,"tag":369,"props":1164,"children":1165},{"style":382},[1166],{"type":64,"value":1167}," \u002Ftmp\u002Fredteam-policy.json",{"type":59,"tag":369,"props":1169,"children":1170},{"style":382},[1171],{"type":64,"value":648},{"type":59,"tag":369,"props":1173,"children":1174},{"style":382},[1175],{"type":64,"value":653},{"type":59,"tag":369,"props":1177,"children":1178},{"style":382},[1179],{"type":64,"value":658},{"type":59,"tag":67,"props":1181,"children":1182},{},[1183,1185,1191],{"type":64,"value":1184},"For error-only reruns that should update the original evaluation in place, use\n",{"type":59,"tag":73,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":64,"value":1190},"promptfoo retry \u003CevalId>",{"type":64,"value":1192}," instead of creating another eval.",{"type":59,"tag":191,"props":1194,"children":1196},{"id":1195},"_6-report-or-gate",[1197],{"type":64,"value":1198},"6. Report or gate",{"type":59,"tag":67,"props":1200,"children":1201},{},[1202,1203,1209],{"type":64,"value":204},{"type":59,"tag":73,"props":1204,"children":1206},{"className":1205},[],[1207],{"type":64,"value":1208},"redteam report",{"type":64,"value":1210}," for interactive triage after results are written. It starts\nor reuses the local Promptfoo UI, so ask before running it unless the user\nexplicitly requested the report UI:",{"type":59,"tag":358,"props":1212,"children":1214},{"className":360,"code":1213,"language":362,"meta":363,"style":363},"npm run local -- redteam report\n",[1215],{"type":59,"tag":73,"props":1216,"children":1217},{"__ignoreMap":363},[1218],{"type":59,"tag":369,"props":1219,"children":1220},{"class":371,"line":372},[1221,1225,1229,1233,1237,1241],{"type":59,"tag":369,"props":1222,"children":1223},{"style":376},[1224],{"type":64,"value":379},{"type":59,"tag":369,"props":1226,"children":1227},{"style":382},[1228],{"type":64,"value":385},{"type":59,"tag":369,"props":1230,"children":1231},{"style":382},[1232],{"type":64,"value":390},{"type":59,"tag":369,"props":1234,"children":1235},{"style":382},[1236],{"type":64,"value":395},{"type":59,"tag":369,"props":1238,"children":1239},{"style":382},[1240],{"type":64,"value":619},{"type":59,"tag":369,"props":1242,"children":1243},{"style":382},[1244],{"type":64,"value":1245}," report\n",{"type":59,"tag":67,"props":1247,"children":1248},{},[1249,1251,1256],{"type":64,"value":1250},"For CI, gate on explicit metrics from the JSON export. Keep thresholds tied to\nthe app's risk tolerance and track category-level changes with\n",{"type":59,"tag":73,"props":1252,"children":1254},{"className":1253},[],[1255],{"type":64,"value":510},{"type":64,"value":351},{"type":59,"tag":103,"props":1258,"children":1260},{"id":1259},"common-mistakes",[1261],{"type":64,"value":1262},"Common Mistakes",{"type":59,"tag":358,"props":1264,"children":1266},{"className":360,"code":1265,"language":362,"meta":363,"style":363},"# WRONG: regenerates probes when you only wanted a comparable rerun\npromptfoo redteam run\n\n# BETTER: reuse generated tests\npromptfoo redteam eval -c redteam.yaml -o results.json --no-cache --no-share\n",[1267],{"type":59,"tag":73,"props":1268,"children":1269},{"__ignoreMap":363},[1270,1278,1294,1303,1311],{"type":59,"tag":369,"props":1271,"children":1272},{"class":371,"line":372},[1273],{"type":59,"tag":369,"props":1274,"children":1275},{"style":459},[1276],{"type":64,"value":1277},"# WRONG: regenerates probes when you only wanted a comparable rerun\n",{"type":59,"tag":369,"props":1279,"children":1280},{"class":371,"line":418},[1281,1285,1289],{"type":59,"tag":369,"props":1282,"children":1283},{"style":376},[1284],{"type":64,"value":8},{"type":59,"tag":369,"props":1286,"children":1287},{"style":382},[1288],{"type":64,"value":619},{"type":59,"tag":369,"props":1290,"children":1291},{"style":382},[1292],{"type":64,"value":1293}," run\n",{"type":59,"tag":369,"props":1295,"children":1296},{"class":371,"line":455},[1297],{"type":59,"tag":369,"props":1298,"children":1300},{"emptyLinePlaceholder":1299},true,[1301],{"type":64,"value":1302},"\n",{"type":59,"tag":369,"props":1304,"children":1305},{"class":371,"line":465},[1306],{"type":59,"tag":369,"props":1307,"children":1308},{"style":459},[1309],{"type":64,"value":1310},"# BETTER: reuse generated tests\n",{"type":59,"tag":369,"props":1312,"children":1314},{"class":371,"line":1313},5,[1315,1319,1323,1327,1331,1336,1340,1345,1349],{"type":59,"tag":369,"props":1316,"children":1317},{"style":376},[1318],{"type":64,"value":8},{"type":59,"tag":369,"props":1320,"children":1321},{"style":382},[1322],{"type":64,"value":619},{"type":59,"tag":369,"props":1324,"children":1325},{"style":382},[1326],{"type":64,"value":624},{"type":59,"tag":369,"props":1328,"children":1329},{"style":382},[1330],{"type":64,"value":410},{"type":59,"tag":369,"props":1332,"children":1333},{"style":382},[1334],{"type":64,"value":1335}," redteam.yaml",{"type":59,"tag":369,"props":1337,"children":1338},{"style":382},[1339],{"type":64,"value":638},{"type":59,"tag":369,"props":1341,"children":1342},{"style":382},[1343],{"type":64,"value":1344}," results.json",{"type":59,"tag":369,"props":1346,"children":1347},{"style":382},[1348],{"type":64,"value":648},{"type":59,"tag":369,"props":1350,"children":1351},{"style":382},[1352],{"type":64,"value":1353}," --no-share\n",{"type":59,"tag":358,"props":1355,"children":1357},{"className":360,"code":1356,"language":362,"meta":363,"style":363},"# WRONG: broad rerun after a flaky target error\npromptfoo redteam eval -c redteam.yaml\n\n# BETTER: rerun only target\u002Fgrader errors from the prior result\npromptfoo redteam eval -c redteam.yaml --filter-errors-only results.json -o errors-rerun.json --no-cache --no-share\n",[1358],{"type":59,"tag":73,"props":1359,"children":1360},{"__ignoreMap":363},[1361,1369,1393,1400,1408],{"type":59,"tag":369,"props":1362,"children":1363},{"class":371,"line":372},[1364],{"type":59,"tag":369,"props":1365,"children":1366},{"style":459},[1367],{"type":64,"value":1368},"# WRONG: broad rerun after a flaky target error\n",{"type":59,"tag":369,"props":1370,"children":1371},{"class":371,"line":418},[1372,1376,1380,1384,1388],{"type":59,"tag":369,"props":1373,"children":1374},{"style":376},[1375],{"type":64,"value":8},{"type":59,"tag":369,"props":1377,"children":1378},{"style":382},[1379],{"type":64,"value":619},{"type":59,"tag":369,"props":1381,"children":1382},{"style":382},[1383],{"type":64,"value":624},{"type":59,"tag":369,"props":1385,"children":1386},{"style":382},[1387],{"type":64,"value":410},{"type":59,"tag":369,"props":1389,"children":1390},{"style":382},[1391],{"type":64,"value":1392}," redteam.yaml\n",{"type":59,"tag":369,"props":1394,"children":1395},{"class":371,"line":455},[1396],{"type":59,"tag":369,"props":1397,"children":1398},{"emptyLinePlaceholder":1299},[1399],{"type":64,"value":1302},{"type":59,"tag":369,"props":1401,"children":1402},{"class":371,"line":465},[1403],{"type":59,"tag":369,"props":1404,"children":1405},{"style":459},[1406],{"type":64,"value":1407},"# BETTER: rerun only target\u002Fgrader errors from the prior result\n",{"type":59,"tag":369,"props":1409,"children":1410},{"class":371,"line":1313},[1411,1415,1419,1423,1427,1431,1435,1439,1443,1448,1452],{"type":59,"tag":369,"props":1412,"children":1413},{"style":376},[1414],{"type":64,"value":8},{"type":59,"tag":369,"props":1416,"children":1417},{"style":382},[1418],{"type":64,"value":619},{"type":59,"tag":369,"props":1420,"children":1421},{"style":382},[1422],{"type":64,"value":624},{"type":59,"tag":369,"props":1424,"children":1425},{"style":382},[1426],{"type":64,"value":410},{"type":59,"tag":369,"props":1428,"children":1429},{"style":382},[1430],{"type":64,"value":1335},{"type":59,"tag":369,"props":1432,"children":1433},{"style":382},[1434],{"type":64,"value":1088},{"type":59,"tag":369,"props":1436,"children":1437},{"style":382},[1438],{"type":64,"value":1344},{"type":59,"tag":369,"props":1440,"children":1441},{"style":382},[1442],{"type":64,"value":638},{"type":59,"tag":369,"props":1444,"children":1445},{"style":382},[1446],{"type":64,"value":1447}," errors-rerun.json",{"type":59,"tag":369,"props":1449,"children":1450},{"style":382},[1451],{"type":64,"value":648},{"type":59,"tag":369,"props":1453,"children":1454},{"style":382},[1455],{"type":64,"value":1353},{"type":59,"tag":103,"props":1457,"children":1459},{"id":1458},"output-contract",[1460],{"type":64,"value":1461},"Output Contract",{"type":59,"tag":67,"props":1463,"children":1464},{},[1465],{"type":64,"value":1466},"When done, state:",{"type":59,"tag":115,"props":1468,"children":1469},{},[1470,1475,1480,1485,1490,1495],{"type":59,"tag":119,"props":1471,"children":1472},{},[1473],{"type":64,"value":1474},"Run mode used: existing generated tests or regenerate-and-run",{"type":59,"tag":119,"props":1476,"children":1477},{},[1478],{"type":64,"value":1479},"Config\u002Fresult\u002Freport paths",{"type":59,"tag":119,"props":1481,"children":1482},{},[1483],{"type":64,"value":1484},"Data-sharing mode and grader\u002Fprovider used",{"type":59,"tag":119,"props":1486,"children":1487},{},[1488],{"type":64,"value":1489},"Commands run and whether any returned nonzero due to redteam failures",{"type":59,"tag":119,"props":1491,"children":1492},{},[1493],{"type":64,"value":1494},"Success, failure, error counts and attack success rate",{"type":59,"tag":119,"props":1496,"children":1497},{},[1498],{"type":64,"value":1499},"Highest-priority failing plugins\u002Fstrategies and recommended next rerun or fix",{"type":59,"tag":1501,"props":1502,"children":1503},"style",{},[1504],{"type":64,"value":1505},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1507,"total":465},[1508,1520,1532,1540],{"slug":1509,"name":1509,"fn":1510,"description":1511,"org":1512,"tags":1513,"stars":25,"repoUrl":26,"updatedAt":1519},"promptfoo-evals","run and refine promptfoo evaluation suites","Write, refine, run, and QA non-redteam promptfoo eval suites after the target or provider already works: prompts, vars, test cases, assertions, model-graded rubrics, transforms, datasets, output exports, filters, and CI gates. Use for regression tests and eval-suite authoring. Do not use for connecting a new target\u002Fprovider, mapping HTTP requests or auth, smoke-testing an endpoint, or redteam plugin\u002Fstrategy setup; use `promptfoo-provider-setup` for connection work instead.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1514,1515,1517,1518],{"name":17,"slug":18,"type":14},{"name":1516,"slug":36,"type":14},"LLM",{"name":8,"slug":8,"type":14},{"name":23,"slug":24,"type":14},"2026-07-18T05:48:36.3451",{"slug":173,"name":173,"fn":1521,"description":1522,"org":1523,"tags":1524,"stars":25,"repoUrl":26,"updatedAt":1531},"configure promptfoo providers and targets","Configure promptfoo providers or redteam targets for hosted models, live HTTP APIs, Python\u002FJavaScript local scripts, agent SDKs, or multi-input systems. Use when connecting promptfoo to the system under test, mapping vars, auth env vars, request bodies, response transforms, or static-code-derived provider wrappers. Do not use for choosing eval assertions or red team plugins unless a smoke test is needed to verify the connection.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1525,1528,1529,1530],{"name":1526,"slug":1527,"type":14},"API Development","api-development",{"name":17,"slug":18,"type":14},{"name":1516,"slug":36,"type":14},{"name":23,"slug":24,"type":14},"2026-07-18T05:47:13.657873",{"slug":4,"name":4,"fn":5,"description":6,"org":1533,"tags":1534,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1535,1536,1537,1538,1539],{"name":20,"slug":21,"type":14},{"name":17,"slug":18,"type":14},{"name":8,"slug":8,"type":14},{"name":12,"slug":13,"type":14},{"name":23,"slug":24,"type":14},{"slug":181,"name":181,"fn":1541,"description":1542,"org":1543,"tags":1544,"stars":25,"repoUrl":26,"updatedAt":1551},"configure promptfoo redteam security setups","Create or refine promptfoo redteam setup configs: purpose, targets, plugins, strategies, frameworks, multi-input target inputs, policy text, grader guidance, contexts, and static-code-derived target\u002Fthreat mapping. Use when preparing a red team scan plan from live probes, code evidence, or provider configs, or when generating adversarial test cases for QA. Do not use for basic provider wiring alone or for running\u002Fevaluating an already-generated redteam scan.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1545,1548,1549,1550],{"name":1546,"slug":1547,"type":14},"Configuration","configuration",{"name":17,"slug":18,"type":14},{"name":8,"slug":8,"type":14},{"name":12,"slug":13,"type":14},"2026-07-18T05:48:36.82567",{"items":1553,"total":465},[1554,1561,1568,1576],{"slug":1509,"name":1509,"fn":1510,"description":1511,"org":1555,"tags":1556,"stars":25,"repoUrl":26,"updatedAt":1519},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1557,1558,1559,1560],{"name":17,"slug":18,"type":14},{"name":1516,"slug":36,"type":14},{"name":8,"slug":8,"type":14},{"name":23,"slug":24,"type":14},{"slug":173,"name":173,"fn":1521,"description":1522,"org":1562,"tags":1563,"stars":25,"repoUrl":26,"updatedAt":1531},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1564,1565,1566,1567],{"name":1526,"slug":1527,"type":14},{"name":17,"slug":18,"type":14},{"name":1516,"slug":36,"type":14},{"name":23,"slug":24,"type":14},{"slug":4,"name":4,"fn":5,"description":6,"org":1569,"tags":1570,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1571,1572,1573,1574,1575],{"name":20,"slug":21,"type":14},{"name":17,"slug":18,"type":14},{"name":8,"slug":8,"type":14},{"name":12,"slug":13,"type":14},{"name":23,"slug":24,"type":14},{"slug":181,"name":181,"fn":1541,"description":1542,"org":1577,"tags":1578,"stars":25,"repoUrl":26,"updatedAt":1551},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1579,1580,1581,1582],{"name":1546,"slug":1547,"type":14},{"name":17,"slug":18,"type":14},{"name":8,"slug":8,"type":14},{"name":12,"slug":13,"type":14}]