[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-promptfoo-promptfoo-evals":3,"mdc--pzvfvo-key":49,"related-repo-promptfoo-promptfoo-evals":1471,"related-org-promptfoo-promptfoo-evals":1519},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":10,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":44,"sourceUrl":47,"mdContent":48},"promptfoo-evals","run and refine promptfoo evaluation suites","Write, refine, run, and QA non-redteam promptfoo eval suites after the target or provider already works: prompts, vars, test cases, assertions, model-graded rubrics, transforms, datasets, output exports, filters, and CI gates. Use for regression tests and eval-suite authoring. Do not use for connecting a new target\u002Fprovider, mapping HTTP requests or auth, smoke-testing an endpoint, or redteam plugin\u002Fstrategy setup; use `promptfoo-provider-setup` for connection work instead.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},"promptfoo","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fpromptfoo.png",[11,13,16,19],{"name":8,"slug":8,"type":12},"tag",{"name":14,"slug":15,"type":12},"LLM","llm",{"name":17,"slug":18,"type":12},"Evals","evals",{"name":20,"slug":21,"type":12},"Testing","testing",23182,"https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo","2026-07-18T05:48:36.3451",null,2079,[28,29,30,31,32,15,33,34,35,36,37,38,39,40,41,42,21,43],"ci","ci-cd","cicd","evaluation","evaluation-framework","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","pentesting","prompt-engineering","prompt-testing","prompts","rag","red-teaming","vulnerability-scanners",{"repoUrl":23,"stars":22,"forks":26,"topics":45,"description":46},[28,29,30,31,32,15,33,34,35,36,37,38,39,40,41,42,21,43],"Test your prompts, agents, and RAGs. Red teaming\u002Fpentesting\u002Fvulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI\u002FCD integration.  Used by OpenAI and Anthropic.","https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo\u002Ftree\u002FHEAD\u002Fplugins\u002Fpromptfoo\u002Fskills\u002Fpromptfoo-evals","---\nname: promptfoo-evals\ndescription: >\n  Write, refine, run, and QA non-redteam promptfoo eval suites after the target\n  or provider already works: prompts, vars, test cases, assertions,\n  model-graded rubrics, transforms, datasets, output exports, filters, and CI\n  gates. Use for regression tests and eval-suite authoring. Do not use for\n  connecting a new target\u002Fprovider, mapping HTTP requests or auth, smoke-testing\n  an endpoint, or redteam plugin\u002Fstrategy setup; use `promptfoo-provider-setup`\n  for connection work instead.\n---\n\n# Promptfoo Evals\n\nBuild a small eval that answers one product question clearly, run it with fresh\nresults, then inspect the exported artifact before expanding.\n\nRead `references\u002Feval-patterns.md` when you need concrete YAML patterns,\nassertion examples, or CI snippets.\nFor deep promptfoo feature questions that are not covered here, consult\n`https:\u002F\u002Fwww.promptfoo.dev\u002Fllms-full.txt`.\n\n## Inputs\n\nInfer these from the repo or user prompt:\n\n- Behavior being evaluated and what \"good\" means.\n- Target\u002Fprovider already configured, or whether `promptfoo-provider-setup` is\n  needed first.\n- Prompt shape and variables.\n- Test data source: inline cases, CSV\u002FJSON, generated data, production examples,\n  or hand-picked regressions.\n- Assertion style: deterministic checks, structured output validation,\n  JavaScript assertions, model-graded rubrics, or a mix.\n- Output needs: JSON export, comparison, CI gate, or human triage.\n\nIf the provider does not work yet, switch to `promptfoo-provider-setup`. If the\ntask is adversarial security scanning, switch to `promptfoo-redteam-setup` or\n`promptfoo-redteam-run`.\n\n## Workflow\n\n### 1. State the eval question\n\nSearch for existing configs first: `promptfooconfig.yaml`,\n`promptfooconfig.yml`, or repo `evals`\u002F`promptfoo` directories. Extend an\nexisting suite when possible.\n\nWrite one sentence for the behavior under test, then choose 3-10 starter cases.\nInclude both ordinary success cases and edge cases that have broken before.\n\nFor new suites, prefer this layout unless the repo already has a convention:\n\n```text\nevals\u002F\u003Csuite-name>\u002F\n  promptfooconfig.yaml\n  prompts\u002F\n  tests\u002F\n```\n\n### 2. Choose assertions\n\nPrefer deterministic assertions first:\n\n- Exact or substring behavior: `equals`, `contains`, `icontains`, `regex`\n- Structured output: `is-json`, `contains-json`, `javascript`\n- Numeric or score-like outputs: `javascript` returning a boolean or score\n- Semantic quality: `llm-rubric` with an explicit grader provider when possible\n\nUse model-graded assertions sparingly for qualities that deterministic checks\ncannot capture. Configure a local or explicit grader for reproducible QA.\n\n### 3. Write the config\n\nInclude:\n\n- `# yaml-language-server: $schema=https:\u002F\u002Fpromptfoo.dev\u002Fconfig-schema.json`\n- A short `description`\n- Field order: `description`, `env`, `prompts`, `providers`, `defaultTest`,\n  `scenarios`, `tests`\n- `prompts` via `file:\u002F\u002Fprompts\u002F*.txt` or `file:\u002F\u002Fprompts\u002F*.json` when prompts\n  are more than a one-line smoke test\n- `tests: file:\u002F\u002Ftests\u002F*.yaml` for suites that will grow beyond a few cases\n- `defaultTest` only for shared assertions\u002Foptions\n- `options.transform` when parsing JSON once makes assertions cleaner\n- Stable metric names only when they help compare dashboards over time\n\nKeep secrets as `{{env.VAR}}`; do not commit `.env` values.\nWhen checking faithfulness or hallucination with `llm-rubric`, inline the source\nmaterial in the rubric via `{{variable}}` so the grader can actually compare.\n\n### 4. Validate and run\n\nFrom the promptfoo repo:\n\n```bash\nsource ~\u002F.nvm\u002Fnvm.sh && nvm use\nnpm run local -- validate config -c path\u002Fto\u002Fpromptfooconfig.yaml\nnpm run local -- eval -c path\u002Fto\u002Fpromptfooconfig.yaml -o \u002Ftmp\u002Feval-results.json --no-cache --no-share\n```\n\nOutside the repo:\n\n```bash\nnpx promptfoo@latest validate config -c path\u002Fto\u002Fpromptfooconfig.yaml\nnpx promptfoo@latest eval -c path\u002Fto\u002Fpromptfooconfig.yaml -o \u002Ftmp\u002Feval-results.json --no-cache --no-share\n```\n\nInspect the output JSON for `results.stats`, `response.output`, `score`,\n`gradingResult.reason`, and `error`.\n\n### 5. Iterate deliberately\n\n- Add cases when a failure represents real expected behavior.\n- Tighten assertions when false positives pass.\n- Use `--filter-pattern`, `--filter-metadata`, or `--filter-failing` for focused\n  reruns.\n- Keep `--no-cache` while developing so you are not validating stale outputs.\n- Use `--no-share` unless the user asks for a shareable URL.\n\n## Common Mistakes\n\n```yaml\n# WRONG: vague rubric with no examples or grader control\n- type: llm-rubric\n  value: Is this good?\n\n# BETTER: concrete success criteria\n- type: llm-rubric\n  value: >-\n    The answer must cite the requested invoice id, state approved\u002Fdenied, and\n    avoid inventing fields not present in the tool result.\n```\n\n```yaml\n# WRONG: unquoted JS expression that starts with [ or { is parsed as YAML flow\n- type: javascript\n  value: ['billing', 'technical'].includes(output.category)\n\n# BETTER: quote any assertion value that begins with [, {, *, &, or !\n- type: javascript\n  value: \"['billing', 'technical'].includes(output.category)\"\n```\n\n```yaml\n# WRONG: inline prompts that contain JSON-like braces are misread as file paths\nprompts:\n  - 'Classify: {{text}}. Return {\"category\": \"...\"} JSON.'\n\n# BETTER: move non-trivial prompts (JSON examples, multi-line, quotes) to a file\nprompts:\n  - file:\u002F\u002F.\u002Fprompts\u002Fclassify.txt\n```\n\n```yaml\n# WRONG: reparsing JSON in every assertion\nassert:\n  - type: javascript\n    value: JSON.parse(output).status === 'approved'\n\n# BETTER: parse once for the test\noptions:\n  transform: JSON.parse(output)\nassert:\n  - type: javascript\n    value: output.status === 'approved'\n```\n\n## Output Contract\n\nWhen done, state:\n\n- Eval question and target\u002Fprovider used\n- Files created or changed\n- Assertion strategy and why\n- Validation\u002Feval commands run\n- Result stats and any failures\u002Ferrors\n- Required environment variables\n- Follow-up cases or assertions to add next\n",{"data":50,"body":51},{"name":4,"description":6},{"type":52,"children":53},"root",[54,62,68,90,97,102,145,172,178,185,220,225,230,242,248,253,339,344,350,355,492,528,534,539,681,686,761,802,808,874,880,1020,1174,1261,1416,1422,1427,1465],{"type":55,"tag":56,"props":57,"children":58},"element","h1",{"id":4},[59],{"type":60,"value":61},"text","Promptfoo Evals",{"type":55,"tag":63,"props":64,"children":65},"p",{},[66],{"type":60,"value":67},"Build a small eval that answers one product question clearly, run it with fresh\nresults, then inspect the exported artifact before expanding.",{"type":55,"tag":63,"props":69,"children":70},{},[71,73,80,82,88],{"type":60,"value":72},"Read ",{"type":55,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":60,"value":79},"references\u002Feval-patterns.md",{"type":60,"value":81}," when you need concrete YAML patterns,\nassertion examples, or CI snippets.\nFor deep promptfoo feature questions that are not covered here, consult\n",{"type":55,"tag":74,"props":83,"children":85},{"className":84},[],[86],{"type":60,"value":87},"https:\u002F\u002Fwww.promptfoo.dev\u002Fllms-full.txt",{"type":60,"value":89},".",{"type":55,"tag":91,"props":92,"children":94},"h2",{"id":93},"inputs",[95],{"type":60,"value":96},"Inputs",{"type":55,"tag":63,"props":98,"children":99},{},[100],{"type":60,"value":101},"Infer these from the repo or user prompt:",{"type":55,"tag":103,"props":104,"children":105},"ul",{},[106,112,125,130,135,140],{"type":55,"tag":107,"props":108,"children":109},"li",{},[110],{"type":60,"value":111},"Behavior being evaluated and what \"good\" means.",{"type":55,"tag":107,"props":113,"children":114},{},[115,117,123],{"type":60,"value":116},"Target\u002Fprovider already configured, or whether ",{"type":55,"tag":74,"props":118,"children":120},{"className":119},[],[121],{"type":60,"value":122},"promptfoo-provider-setup",{"type":60,"value":124}," is\nneeded first.",{"type":55,"tag":107,"props":126,"children":127},{},[128],{"type":60,"value":129},"Prompt shape and variables.",{"type":55,"tag":107,"props":131,"children":132},{},[133],{"type":60,"value":134},"Test data source: inline cases, CSV\u002FJSON, generated data, production examples,\nor hand-picked regressions.",{"type":55,"tag":107,"props":136,"children":137},{},[138],{"type":60,"value":139},"Assertion style: deterministic checks, structured output validation,\nJavaScript assertions, model-graded rubrics, or a mix.",{"type":55,"tag":107,"props":141,"children":142},{},[143],{"type":60,"value":144},"Output needs: JSON export, comparison, CI gate, or human triage.",{"type":55,"tag":63,"props":146,"children":147},{},[148,150,155,157,163,165,171],{"type":60,"value":149},"If the provider does not work yet, switch to ",{"type":55,"tag":74,"props":151,"children":153},{"className":152},[],[154],{"type":60,"value":122},{"type":60,"value":156},". If the\ntask is adversarial security scanning, switch to ",{"type":55,"tag":74,"props":158,"children":160},{"className":159},[],[161],{"type":60,"value":162},"promptfoo-redteam-setup",{"type":60,"value":164}," or\n",{"type":55,"tag":74,"props":166,"children":168},{"className":167},[],[169],{"type":60,"value":170},"promptfoo-redteam-run",{"type":60,"value":89},{"type":55,"tag":91,"props":173,"children":175},{"id":174},"workflow",[176],{"type":60,"value":177},"Workflow",{"type":55,"tag":179,"props":180,"children":182},"h3",{"id":181},"_1-state-the-eval-question",[183],{"type":60,"value":184},"1. State the eval question",{"type":55,"tag":63,"props":186,"children":187},{},[188,190,196,198,204,206,211,213,218],{"type":60,"value":189},"Search for existing configs first: ",{"type":55,"tag":74,"props":191,"children":193},{"className":192},[],[194],{"type":60,"value":195},"promptfooconfig.yaml",{"type":60,"value":197},",\n",{"type":55,"tag":74,"props":199,"children":201},{"className":200},[],[202],{"type":60,"value":203},"promptfooconfig.yml",{"type":60,"value":205},", or repo ",{"type":55,"tag":74,"props":207,"children":209},{"className":208},[],[210],{"type":60,"value":18},{"type":60,"value":212},"\u002F",{"type":55,"tag":74,"props":214,"children":216},{"className":215},[],[217],{"type":60,"value":8},{"type":60,"value":219}," directories. Extend an\nexisting suite when possible.",{"type":55,"tag":63,"props":221,"children":222},{},[223],{"type":60,"value":224},"Write one sentence for the behavior under test, then choose 3-10 starter cases.\nInclude both ordinary success cases and edge cases that have broken before.",{"type":55,"tag":63,"props":226,"children":227},{},[228],{"type":60,"value":229},"For new suites, prefer this layout unless the repo already has a convention:",{"type":55,"tag":231,"props":232,"children":237},"pre",{"className":233,"code":235,"language":60,"meta":236},[234],"language-text","evals\u002F\u003Csuite-name>\u002F\n  promptfooconfig.yaml\n  prompts\u002F\n  tests\u002F\n","",[238],{"type":55,"tag":74,"props":239,"children":240},{"__ignoreMap":236},[241],{"type":60,"value":235},{"type":55,"tag":179,"props":243,"children":245},{"id":244},"_2-choose-assertions",[246],{"type":60,"value":247},"2. Choose assertions",{"type":55,"tag":63,"props":249,"children":250},{},[251],{"type":60,"value":252},"Prefer deterministic assertions first:",{"type":55,"tag":103,"props":254,"children":255},{},[256,289,314,326],{"type":55,"tag":107,"props":257,"children":258},{},[259,261,267,269,275,276,282,283],{"type":60,"value":260},"Exact or substring behavior: ",{"type":55,"tag":74,"props":262,"children":264},{"className":263},[],[265],{"type":60,"value":266},"equals",{"type":60,"value":268},", ",{"type":55,"tag":74,"props":270,"children":272},{"className":271},[],[273],{"type":60,"value":274},"contains",{"type":60,"value":268},{"type":55,"tag":74,"props":277,"children":279},{"className":278},[],[280],{"type":60,"value":281},"icontains",{"type":60,"value":268},{"type":55,"tag":74,"props":284,"children":286},{"className":285},[],[287],{"type":60,"value":288},"regex",{"type":55,"tag":107,"props":290,"children":291},{},[292,294,300,301,307,308],{"type":60,"value":293},"Structured output: ",{"type":55,"tag":74,"props":295,"children":297},{"className":296},[],[298],{"type":60,"value":299},"is-json",{"type":60,"value":268},{"type":55,"tag":74,"props":302,"children":304},{"className":303},[],[305],{"type":60,"value":306},"contains-json",{"type":60,"value":268},{"type":55,"tag":74,"props":309,"children":311},{"className":310},[],[312],{"type":60,"value":313},"javascript",{"type":55,"tag":107,"props":315,"children":316},{},[317,319,324],{"type":60,"value":318},"Numeric or score-like outputs: ",{"type":55,"tag":74,"props":320,"children":322},{"className":321},[],[323],{"type":60,"value":313},{"type":60,"value":325}," returning a boolean or score",{"type":55,"tag":107,"props":327,"children":328},{},[329,331,337],{"type":60,"value":330},"Semantic quality: ",{"type":55,"tag":74,"props":332,"children":334},{"className":333},[],[335],{"type":60,"value":336},"llm-rubric",{"type":60,"value":338}," with an explicit grader provider when possible",{"type":55,"tag":63,"props":340,"children":341},{},[342],{"type":60,"value":343},"Use model-graded assertions sparingly for qualities that deterministic checks\ncannot capture. Configure a local or explicit grader for reproducible QA.",{"type":55,"tag":179,"props":345,"children":347},{"id":346},"_3-write-the-config",[348],{"type":60,"value":349},"3. Write the config",{"type":55,"tag":63,"props":351,"children":352},{},[353],{"type":60,"value":354},"Include:",{"type":55,"tag":103,"props":356,"children":357},{},[358,367,378,429,455,466,476,487],{"type":55,"tag":107,"props":359,"children":360},{},[361],{"type":55,"tag":74,"props":362,"children":364},{"className":363},[],[365],{"type":60,"value":366},"# yaml-language-server: $schema=https:\u002F\u002Fpromptfoo.dev\u002Fconfig-schema.json",{"type":55,"tag":107,"props":368,"children":369},{},[370,372],{"type":60,"value":371},"A short ",{"type":55,"tag":74,"props":373,"children":375},{"className":374},[],[376],{"type":60,"value":377},"description",{"type":55,"tag":107,"props":379,"children":380},{},[381,383,388,389,395,396,401,402,408,409,415,416,422,423],{"type":60,"value":382},"Field order: ",{"type":55,"tag":74,"props":384,"children":386},{"className":385},[],[387],{"type":60,"value":377},{"type":60,"value":268},{"type":55,"tag":74,"props":390,"children":392},{"className":391},[],[393],{"type":60,"value":394},"env",{"type":60,"value":268},{"type":55,"tag":74,"props":397,"children":399},{"className":398},[],[400],{"type":60,"value":40},{"type":60,"value":268},{"type":55,"tag":74,"props":403,"children":405},{"className":404},[],[406],{"type":60,"value":407},"providers",{"type":60,"value":268},{"type":55,"tag":74,"props":410,"children":412},{"className":411},[],[413],{"type":60,"value":414},"defaultTest",{"type":60,"value":197},{"type":55,"tag":74,"props":417,"children":419},{"className":418},[],[420],{"type":60,"value":421},"scenarios",{"type":60,"value":268},{"type":55,"tag":74,"props":424,"children":426},{"className":425},[],[427],{"type":60,"value":428},"tests",{"type":55,"tag":107,"props":430,"children":431},{},[432,437,439,445,447,453],{"type":55,"tag":74,"props":433,"children":435},{"className":434},[],[436],{"type":60,"value":40},{"type":60,"value":438}," via ",{"type":55,"tag":74,"props":440,"children":442},{"className":441},[],[443],{"type":60,"value":444},"file:\u002F\u002Fprompts\u002F*.txt",{"type":60,"value":446}," or ",{"type":55,"tag":74,"props":448,"children":450},{"className":449},[],[451],{"type":60,"value":452},"file:\u002F\u002Fprompts\u002F*.json",{"type":60,"value":454}," when prompts\nare more than a one-line smoke test",{"type":55,"tag":107,"props":456,"children":457},{},[458,464],{"type":55,"tag":74,"props":459,"children":461},{"className":460},[],[462],{"type":60,"value":463},"tests: file:\u002F\u002Ftests\u002F*.yaml",{"type":60,"value":465}," for suites that will grow beyond a few cases",{"type":55,"tag":107,"props":467,"children":468},{},[469,474],{"type":55,"tag":74,"props":470,"children":472},{"className":471},[],[473],{"type":60,"value":414},{"type":60,"value":475}," only for shared assertions\u002Foptions",{"type":55,"tag":107,"props":477,"children":478},{},[479,485],{"type":55,"tag":74,"props":480,"children":482},{"className":481},[],[483],{"type":60,"value":484},"options.transform",{"type":60,"value":486}," when parsing JSON once makes assertions cleaner",{"type":55,"tag":107,"props":488,"children":489},{},[490],{"type":60,"value":491},"Stable metric names only when they help compare dashboards over time",{"type":55,"tag":63,"props":493,"children":494},{},[495,497,503,505,511,513,518,520,526],{"type":60,"value":496},"Keep secrets as ",{"type":55,"tag":74,"props":498,"children":500},{"className":499},[],[501],{"type":60,"value":502},"{{env.VAR}}",{"type":60,"value":504},"; do not commit ",{"type":55,"tag":74,"props":506,"children":508},{"className":507},[],[509],{"type":60,"value":510},".env",{"type":60,"value":512}," values.\nWhen checking faithfulness or hallucination with ",{"type":55,"tag":74,"props":514,"children":516},{"className":515},[],[517],{"type":60,"value":336},{"type":60,"value":519},", inline the source\nmaterial in the rubric via ",{"type":55,"tag":74,"props":521,"children":523},{"className":522},[],[524],{"type":60,"value":525},"{{variable}}",{"type":60,"value":527}," so the grader can actually compare.",{"type":55,"tag":179,"props":529,"children":531},{"id":530},"_4-validate-and-run",[532],{"type":60,"value":533},"4. Validate and run",{"type":55,"tag":63,"props":535,"children":536},{},[537],{"type":60,"value":538},"From the promptfoo repo:",{"type":55,"tag":231,"props":540,"children":544},{"className":541,"code":542,"language":543,"meta":236,"style":236},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","source ~\u002F.nvm\u002Fnvm.sh && nvm use\nnpm run local -- validate config -c path\u002Fto\u002Fpromptfooconfig.yaml\nnpm run local -- eval -c path\u002Fto\u002Fpromptfooconfig.yaml -o \u002Ftmp\u002Feval-results.json --no-cache --no-share\n","bash",[545],{"type":55,"tag":74,"props":546,"children":547},{"__ignoreMap":236},[548,583,627],{"type":55,"tag":549,"props":550,"children":553},"span",{"class":551,"line":552},"line",1,[554,560,566,572,578],{"type":55,"tag":549,"props":555,"children":557},{"style":556},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[558],{"type":60,"value":559},"source",{"type":55,"tag":549,"props":561,"children":563},{"style":562},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[564],{"type":60,"value":565}," ~\u002F.nvm\u002Fnvm.sh",{"type":55,"tag":549,"props":567,"children":569},{"style":568},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[570],{"type":60,"value":571}," &&",{"type":55,"tag":549,"props":573,"children":575},{"style":574},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[576],{"type":60,"value":577}," nvm",{"type":55,"tag":549,"props":579,"children":580},{"style":562},[581],{"type":60,"value":582}," use\n",{"type":55,"tag":549,"props":584,"children":586},{"class":551,"line":585},2,[587,592,597,602,607,612,617,622],{"type":55,"tag":549,"props":588,"children":589},{"style":574},[590],{"type":60,"value":591},"npm",{"type":55,"tag":549,"props":593,"children":594},{"style":562},[595],{"type":60,"value":596}," run",{"type":55,"tag":549,"props":598,"children":599},{"style":562},[600],{"type":60,"value":601}," local",{"type":55,"tag":549,"props":603,"children":604},{"style":562},[605],{"type":60,"value":606}," --",{"type":55,"tag":549,"props":608,"children":609},{"style":562},[610],{"type":60,"value":611}," validate",{"type":55,"tag":549,"props":613,"children":614},{"style":562},[615],{"type":60,"value":616}," config",{"type":55,"tag":549,"props":618,"children":619},{"style":562},[620],{"type":60,"value":621}," -c",{"type":55,"tag":549,"props":623,"children":624},{"style":562},[625],{"type":60,"value":626}," path\u002Fto\u002Fpromptfooconfig.yaml\n",{"type":55,"tag":549,"props":628,"children":630},{"class":551,"line":629},3,[631,635,639,643,647,652,656,661,666,671,676],{"type":55,"tag":549,"props":632,"children":633},{"style":574},[634],{"type":60,"value":591},{"type":55,"tag":549,"props":636,"children":637},{"style":562},[638],{"type":60,"value":596},{"type":55,"tag":549,"props":640,"children":641},{"style":562},[642],{"type":60,"value":601},{"type":55,"tag":549,"props":644,"children":645},{"style":562},[646],{"type":60,"value":606},{"type":55,"tag":549,"props":648,"children":649},{"style":562},[650],{"type":60,"value":651}," eval",{"type":55,"tag":549,"props":653,"children":654},{"style":562},[655],{"type":60,"value":621},{"type":55,"tag":549,"props":657,"children":658},{"style":562},[659],{"type":60,"value":660}," path\u002Fto\u002Fpromptfooconfig.yaml",{"type":55,"tag":549,"props":662,"children":663},{"style":562},[664],{"type":60,"value":665}," -o",{"type":55,"tag":549,"props":667,"children":668},{"style":562},[669],{"type":60,"value":670}," \u002Ftmp\u002Feval-results.json",{"type":55,"tag":549,"props":672,"children":673},{"style":562},[674],{"type":60,"value":675}," --no-cache",{"type":55,"tag":549,"props":677,"children":678},{"style":562},[679],{"type":60,"value":680}," --no-share\n",{"type":55,"tag":63,"props":682,"children":683},{},[684],{"type":60,"value":685},"Outside the repo:",{"type":55,"tag":231,"props":687,"children":689},{"className":541,"code":688,"language":543,"meta":236,"style":236},"npx promptfoo@latest validate config -c path\u002Fto\u002Fpromptfooconfig.yaml\nnpx promptfoo@latest eval -c path\u002Fto\u002Fpromptfooconfig.yaml -o \u002Ftmp\u002Feval-results.json --no-cache --no-share\n",[690],{"type":55,"tag":74,"props":691,"children":692},{"__ignoreMap":236},[693,722],{"type":55,"tag":549,"props":694,"children":695},{"class":551,"line":552},[696,701,706,710,714,718],{"type":55,"tag":549,"props":697,"children":698},{"style":574},[699],{"type":60,"value":700},"npx",{"type":55,"tag":549,"props":702,"children":703},{"style":562},[704],{"type":60,"value":705}," promptfoo@latest",{"type":55,"tag":549,"props":707,"children":708},{"style":562},[709],{"type":60,"value":611},{"type":55,"tag":549,"props":711,"children":712},{"style":562},[713],{"type":60,"value":616},{"type":55,"tag":549,"props":715,"children":716},{"style":562},[717],{"type":60,"value":621},{"type":55,"tag":549,"props":719,"children":720},{"style":562},[721],{"type":60,"value":626},{"type":55,"tag":549,"props":723,"children":724},{"class":551,"line":585},[725,729,733,737,741,745,749,753,757],{"type":55,"tag":549,"props":726,"children":727},{"style":574},[728],{"type":60,"value":700},{"type":55,"tag":549,"props":730,"children":731},{"style":562},[732],{"type":60,"value":705},{"type":55,"tag":549,"props":734,"children":735},{"style":562},[736],{"type":60,"value":651},{"type":55,"tag":549,"props":738,"children":739},{"style":562},[740],{"type":60,"value":621},{"type":55,"tag":549,"props":742,"children":743},{"style":562},[744],{"type":60,"value":660},{"type":55,"tag":549,"props":746,"children":747},{"style":562},[748],{"type":60,"value":665},{"type":55,"tag":549,"props":750,"children":751},{"style":562},[752],{"type":60,"value":670},{"type":55,"tag":549,"props":754,"children":755},{"style":562},[756],{"type":60,"value":675},{"type":55,"tag":549,"props":758,"children":759},{"style":562},[760],{"type":60,"value":680},{"type":55,"tag":63,"props":762,"children":763},{},[764,766,772,773,779,780,786,787,793,795,801],{"type":60,"value":765},"Inspect the output JSON for ",{"type":55,"tag":74,"props":767,"children":769},{"className":768},[],[770],{"type":60,"value":771},"results.stats",{"type":60,"value":268},{"type":55,"tag":74,"props":774,"children":776},{"className":775},[],[777],{"type":60,"value":778},"response.output",{"type":60,"value":268},{"type":55,"tag":74,"props":781,"children":783},{"className":782},[],[784],{"type":60,"value":785},"score",{"type":60,"value":197},{"type":55,"tag":74,"props":788,"children":790},{"className":789},[],[791],{"type":60,"value":792},"gradingResult.reason",{"type":60,"value":794},", and ",{"type":55,"tag":74,"props":796,"children":798},{"className":797},[],[799],{"type":60,"value":800},"error",{"type":60,"value":89},{"type":55,"tag":179,"props":803,"children":805},{"id":804},"_5-iterate-deliberately",[806],{"type":60,"value":807},"5. Iterate deliberately",{"type":55,"tag":103,"props":809,"children":810},{},[811,816,821,849,862],{"type":55,"tag":107,"props":812,"children":813},{},[814],{"type":60,"value":815},"Add cases when a failure represents real expected behavior.",{"type":55,"tag":107,"props":817,"children":818},{},[819],{"type":60,"value":820},"Tighten assertions when false positives pass.",{"type":55,"tag":107,"props":822,"children":823},{},[824,826,832,833,839,841,847],{"type":60,"value":825},"Use ",{"type":55,"tag":74,"props":827,"children":829},{"className":828},[],[830],{"type":60,"value":831},"--filter-pattern",{"type":60,"value":268},{"type":55,"tag":74,"props":834,"children":836},{"className":835},[],[837],{"type":60,"value":838},"--filter-metadata",{"type":60,"value":840},", or ",{"type":55,"tag":74,"props":842,"children":844},{"className":843},[],[845],{"type":60,"value":846},"--filter-failing",{"type":60,"value":848}," for focused\nreruns.",{"type":55,"tag":107,"props":850,"children":851},{},[852,854,860],{"type":60,"value":853},"Keep ",{"type":55,"tag":74,"props":855,"children":857},{"className":856},[],[858],{"type":60,"value":859},"--no-cache",{"type":60,"value":861}," while developing so you are not validating stale outputs.",{"type":55,"tag":107,"props":863,"children":864},{},[865,866,872],{"type":60,"value":825},{"type":55,"tag":74,"props":867,"children":869},{"className":868},[],[870],{"type":60,"value":871},"--no-share",{"type":60,"value":873}," unless the user asks for a shareable URL.",{"type":55,"tag":91,"props":875,"children":877},{"id":876},"common-mistakes",[878],{"type":60,"value":879},"Common Mistakes",{"type":55,"tag":231,"props":881,"children":885},{"className":882,"code":883,"language":884,"meta":236,"style":236},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# WRONG: vague rubric with no examples or grader control\n- type: llm-rubric\n  value: Is this good?\n\n# BETTER: concrete success criteria\n- type: llm-rubric\n  value: >-\n    The answer must cite the requested invoice id, state approved\u002Fdenied, and\n    avoid inventing fields not present in the tool result.\n","yaml",[886],{"type":55,"tag":74,"props":887,"children":888},{"__ignoreMap":236},[889,898,922,939,949,958,978,1002,1011],{"type":55,"tag":549,"props":890,"children":891},{"class":551,"line":552},[892],{"type":55,"tag":549,"props":893,"children":895},{"style":894},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[896],{"type":60,"value":897},"# WRONG: vague rubric with no examples or grader control\n",{"type":55,"tag":549,"props":899,"children":900},{"class":551,"line":585},[901,906,912,917],{"type":55,"tag":549,"props":902,"children":903},{"style":568},[904],{"type":60,"value":905},"-",{"type":55,"tag":549,"props":907,"children":909},{"style":908},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[910],{"type":60,"value":911}," type",{"type":55,"tag":549,"props":913,"children":914},{"style":568},[915],{"type":60,"value":916},":",{"type":55,"tag":549,"props":918,"children":919},{"style":562},[920],{"type":60,"value":921}," llm-rubric\n",{"type":55,"tag":549,"props":923,"children":924},{"class":551,"line":629},[925,930,934],{"type":55,"tag":549,"props":926,"children":927},{"style":908},[928],{"type":60,"value":929},"  value",{"type":55,"tag":549,"props":931,"children":932},{"style":568},[933],{"type":60,"value":916},{"type":55,"tag":549,"props":935,"children":936},{"style":562},[937],{"type":60,"value":938}," Is this good?\n",{"type":55,"tag":549,"props":940,"children":942},{"class":551,"line":941},4,[943],{"type":55,"tag":549,"props":944,"children":946},{"emptyLinePlaceholder":945},true,[947],{"type":60,"value":948},"\n",{"type":55,"tag":549,"props":950,"children":952},{"class":551,"line":951},5,[953],{"type":55,"tag":549,"props":954,"children":955},{"style":894},[956],{"type":60,"value":957},"# BETTER: concrete success criteria\n",{"type":55,"tag":549,"props":959,"children":961},{"class":551,"line":960},6,[962,966,970,974],{"type":55,"tag":549,"props":963,"children":964},{"style":568},[965],{"type":60,"value":905},{"type":55,"tag":549,"props":967,"children":968},{"style":908},[969],{"type":60,"value":911},{"type":55,"tag":549,"props":971,"children":972},{"style":568},[973],{"type":60,"value":916},{"type":55,"tag":549,"props":975,"children":976},{"style":562},[977],{"type":60,"value":921},{"type":55,"tag":549,"props":979,"children":981},{"class":551,"line":980},7,[982,986,990,996],{"type":55,"tag":549,"props":983,"children":984},{"style":908},[985],{"type":60,"value":929},{"type":55,"tag":549,"props":987,"children":988},{"style":568},[989],{"type":60,"value":916},{"type":55,"tag":549,"props":991,"children":993},{"style":992},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[994],{"type":60,"value":995}," >",{"type":55,"tag":549,"props":997,"children":999},{"style":998},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1000],{"type":60,"value":1001},"-\n",{"type":55,"tag":549,"props":1003,"children":1005},{"class":551,"line":1004},8,[1006],{"type":55,"tag":549,"props":1007,"children":1008},{"style":562},[1009],{"type":60,"value":1010},"    The answer must cite the requested invoice id, state approved\u002Fdenied, and\n",{"type":55,"tag":549,"props":1012,"children":1014},{"class":551,"line":1013},9,[1015],{"type":55,"tag":549,"props":1016,"children":1017},{"style":562},[1018],{"type":60,"value":1019},"    avoid inventing fields not present in the tool result.\n",{"type":55,"tag":231,"props":1021,"children":1023},{"className":882,"code":1022,"language":884,"meta":236,"style":236},"# WRONG: unquoted JS expression that starts with [ or { is parsed as YAML flow\n- type: javascript\n  value: ['billing', 'technical'].includes(output.category)\n\n# BETTER: quote any assertion value that begins with [, {, *, &, or !\n- type: javascript\n  value: \"['billing', 'technical'].includes(output.category)\"\n",[1024],{"type":55,"tag":74,"props":1025,"children":1026},{"__ignoreMap":236},[1027,1035,1055,1114,1121,1129,1148],{"type":55,"tag":549,"props":1028,"children":1029},{"class":551,"line":552},[1030],{"type":55,"tag":549,"props":1031,"children":1032},{"style":894},[1033],{"type":60,"value":1034},"# WRONG: unquoted JS expression that starts with [ or { is parsed as YAML flow\n",{"type":55,"tag":549,"props":1036,"children":1037},{"class":551,"line":585},[1038,1042,1046,1050],{"type":55,"tag":549,"props":1039,"children":1040},{"style":568},[1041],{"type":60,"value":905},{"type":55,"tag":549,"props":1043,"children":1044},{"style":908},[1045],{"type":60,"value":911},{"type":55,"tag":549,"props":1047,"children":1048},{"style":568},[1049],{"type":60,"value":916},{"type":55,"tag":549,"props":1051,"children":1052},{"style":562},[1053],{"type":60,"value":1054}," javascript\n",{"type":55,"tag":549,"props":1056,"children":1057},{"class":551,"line":629},[1058,1062,1066,1071,1076,1081,1085,1090,1095,1100,1104,1109],{"type":55,"tag":549,"props":1059,"children":1060},{"style":908},[1061],{"type":60,"value":929},{"type":55,"tag":549,"props":1063,"children":1064},{"style":568},[1065],{"type":60,"value":916},{"type":55,"tag":549,"props":1067,"children":1068},{"style":568},[1069],{"type":60,"value":1070}," [",{"type":55,"tag":549,"props":1072,"children":1073},{"style":568},[1074],{"type":60,"value":1075},"'",{"type":55,"tag":549,"props":1077,"children":1078},{"style":562},[1079],{"type":60,"value":1080},"billing",{"type":55,"tag":549,"props":1082,"children":1083},{"style":568},[1084],{"type":60,"value":1075},{"type":55,"tag":549,"props":1086,"children":1087},{"style":568},[1088],{"type":60,"value":1089},",",{"type":55,"tag":549,"props":1091,"children":1092},{"style":568},[1093],{"type":60,"value":1094}," '",{"type":55,"tag":549,"props":1096,"children":1097},{"style":562},[1098],{"type":60,"value":1099},"technical",{"type":55,"tag":549,"props":1101,"children":1102},{"style":568},[1103],{"type":60,"value":1075},{"type":55,"tag":549,"props":1105,"children":1106},{"style":568},[1107],{"type":60,"value":1108},"]",{"type":55,"tag":549,"props":1110,"children":1111},{"style":562},[1112],{"type":60,"value":1113},".includes(output.category)\n",{"type":55,"tag":549,"props":1115,"children":1116},{"class":551,"line":941},[1117],{"type":55,"tag":549,"props":1118,"children":1119},{"emptyLinePlaceholder":945},[1120],{"type":60,"value":948},{"type":55,"tag":549,"props":1122,"children":1123},{"class":551,"line":951},[1124],{"type":55,"tag":549,"props":1125,"children":1126},{"style":894},[1127],{"type":60,"value":1128},"# BETTER: quote any assertion value that begins with [, {, *, &, or !\n",{"type":55,"tag":549,"props":1130,"children":1131},{"class":551,"line":960},[1132,1136,1140,1144],{"type":55,"tag":549,"props":1133,"children":1134},{"style":568},[1135],{"type":60,"value":905},{"type":55,"tag":549,"props":1137,"children":1138},{"style":908},[1139],{"type":60,"value":911},{"type":55,"tag":549,"props":1141,"children":1142},{"style":568},[1143],{"type":60,"value":916},{"type":55,"tag":549,"props":1145,"children":1146},{"style":562},[1147],{"type":60,"value":1054},{"type":55,"tag":549,"props":1149,"children":1150},{"class":551,"line":980},[1151,1155,1159,1164,1169],{"type":55,"tag":549,"props":1152,"children":1153},{"style":908},[1154],{"type":60,"value":929},{"type":55,"tag":549,"props":1156,"children":1157},{"style":568},[1158],{"type":60,"value":916},{"type":55,"tag":549,"props":1160,"children":1161},{"style":568},[1162],{"type":60,"value":1163}," \"",{"type":55,"tag":549,"props":1165,"children":1166},{"style":562},[1167],{"type":60,"value":1168},"['billing', 'technical'].includes(output.category)",{"type":55,"tag":549,"props":1170,"children":1171},{"style":568},[1172],{"type":60,"value":1173},"\"\n",{"type":55,"tag":231,"props":1175,"children":1177},{"className":882,"code":1176,"language":884,"meta":236,"style":236},"# WRONG: inline prompts that contain JSON-like braces are misread as file paths\nprompts:\n  - 'Classify: {{text}}. Return {\"category\": \"...\"} JSON.'\n\n# BETTER: move non-trivial prompts (JSON examples, multi-line, quotes) to a file\nprompts:\n  - file:\u002F\u002F.\u002Fprompts\u002Fclassify.txt\n",[1178],{"type":55,"tag":74,"props":1179,"children":1180},{"__ignoreMap":236},[1181,1189,1201,1223,1230,1238,1249],{"type":55,"tag":549,"props":1182,"children":1183},{"class":551,"line":552},[1184],{"type":55,"tag":549,"props":1185,"children":1186},{"style":894},[1187],{"type":60,"value":1188},"# WRONG: inline prompts that contain JSON-like braces are misread as file paths\n",{"type":55,"tag":549,"props":1190,"children":1191},{"class":551,"line":585},[1192,1196],{"type":55,"tag":549,"props":1193,"children":1194},{"style":908},[1195],{"type":60,"value":40},{"type":55,"tag":549,"props":1197,"children":1198},{"style":568},[1199],{"type":60,"value":1200},":\n",{"type":55,"tag":549,"props":1202,"children":1203},{"class":551,"line":629},[1204,1209,1213,1218],{"type":55,"tag":549,"props":1205,"children":1206},{"style":568},[1207],{"type":60,"value":1208},"  -",{"type":55,"tag":549,"props":1210,"children":1211},{"style":568},[1212],{"type":60,"value":1094},{"type":55,"tag":549,"props":1214,"children":1215},{"style":562},[1216],{"type":60,"value":1217},"Classify: {{text}}. Return {\"category\": \"...\"} JSON.",{"type":55,"tag":549,"props":1219,"children":1220},{"style":568},[1221],{"type":60,"value":1222},"'\n",{"type":55,"tag":549,"props":1224,"children":1225},{"class":551,"line":941},[1226],{"type":55,"tag":549,"props":1227,"children":1228},{"emptyLinePlaceholder":945},[1229],{"type":60,"value":948},{"type":55,"tag":549,"props":1231,"children":1232},{"class":551,"line":951},[1233],{"type":55,"tag":549,"props":1234,"children":1235},{"style":894},[1236],{"type":60,"value":1237},"# BETTER: move non-trivial prompts (JSON examples, multi-line, quotes) to a file\n",{"type":55,"tag":549,"props":1239,"children":1240},{"class":551,"line":960},[1241,1245],{"type":55,"tag":549,"props":1242,"children":1243},{"style":908},[1244],{"type":60,"value":40},{"type":55,"tag":549,"props":1246,"children":1247},{"style":568},[1248],{"type":60,"value":1200},{"type":55,"tag":549,"props":1250,"children":1251},{"class":551,"line":980},[1252,1256],{"type":55,"tag":549,"props":1253,"children":1254},{"style":568},[1255],{"type":60,"value":1208},{"type":55,"tag":549,"props":1257,"children":1258},{"style":562},[1259],{"type":60,"value":1260}," file:\u002F\u002F.\u002Fprompts\u002Fclassify.txt\n",{"type":55,"tag":231,"props":1262,"children":1264},{"className":882,"code":1263,"language":884,"meta":236,"style":236},"# WRONG: reparsing JSON in every assertion\nassert:\n  - type: javascript\n    value: JSON.parse(output).status === 'approved'\n\n# BETTER: parse once for the test\noptions:\n  transform: JSON.parse(output)\nassert:\n  - type: javascript\n    value: output.status === 'approved'\n",[1265],{"type":55,"tag":74,"props":1266,"children":1267},{"__ignoreMap":236},[1268,1276,1288,1307,1324,1331,1339,1351,1368,1379,1399],{"type":55,"tag":549,"props":1269,"children":1270},{"class":551,"line":552},[1271],{"type":55,"tag":549,"props":1272,"children":1273},{"style":894},[1274],{"type":60,"value":1275},"# WRONG: reparsing JSON in every assertion\n",{"type":55,"tag":549,"props":1277,"children":1278},{"class":551,"line":585},[1279,1284],{"type":55,"tag":549,"props":1280,"children":1281},{"style":908},[1282],{"type":60,"value":1283},"assert",{"type":55,"tag":549,"props":1285,"children":1286},{"style":568},[1287],{"type":60,"value":1200},{"type":55,"tag":549,"props":1289,"children":1290},{"class":551,"line":629},[1291,1295,1299,1303],{"type":55,"tag":549,"props":1292,"children":1293},{"style":568},[1294],{"type":60,"value":1208},{"type":55,"tag":549,"props":1296,"children":1297},{"style":908},[1298],{"type":60,"value":911},{"type":55,"tag":549,"props":1300,"children":1301},{"style":568},[1302],{"type":60,"value":916},{"type":55,"tag":549,"props":1304,"children":1305},{"style":562},[1306],{"type":60,"value":1054},{"type":55,"tag":549,"props":1308,"children":1309},{"class":551,"line":941},[1310,1315,1319],{"type":55,"tag":549,"props":1311,"children":1312},{"style":908},[1313],{"type":60,"value":1314},"    value",{"type":55,"tag":549,"props":1316,"children":1317},{"style":568},[1318],{"type":60,"value":916},{"type":55,"tag":549,"props":1320,"children":1321},{"style":562},[1322],{"type":60,"value":1323}," JSON.parse(output).status === 'approved'\n",{"type":55,"tag":549,"props":1325,"children":1326},{"class":551,"line":951},[1327],{"type":55,"tag":549,"props":1328,"children":1329},{"emptyLinePlaceholder":945},[1330],{"type":60,"value":948},{"type":55,"tag":549,"props":1332,"children":1333},{"class":551,"line":960},[1334],{"type":55,"tag":549,"props":1335,"children":1336},{"style":894},[1337],{"type":60,"value":1338},"# BETTER: parse once for the test\n",{"type":55,"tag":549,"props":1340,"children":1341},{"class":551,"line":980},[1342,1347],{"type":55,"tag":549,"props":1343,"children":1344},{"style":908},[1345],{"type":60,"value":1346},"options",{"type":55,"tag":549,"props":1348,"children":1349},{"style":568},[1350],{"type":60,"value":1200},{"type":55,"tag":549,"props":1352,"children":1353},{"class":551,"line":1004},[1354,1359,1363],{"type":55,"tag":549,"props":1355,"children":1356},{"style":908},[1357],{"type":60,"value":1358},"  transform",{"type":55,"tag":549,"props":1360,"children":1361},{"style":568},[1362],{"type":60,"value":916},{"type":55,"tag":549,"props":1364,"children":1365},{"style":562},[1366],{"type":60,"value":1367}," JSON.parse(output)\n",{"type":55,"tag":549,"props":1369,"children":1370},{"class":551,"line":1013},[1371,1375],{"type":55,"tag":549,"props":1372,"children":1373},{"style":908},[1374],{"type":60,"value":1283},{"type":55,"tag":549,"props":1376,"children":1377},{"style":568},[1378],{"type":60,"value":1200},{"type":55,"tag":549,"props":1380,"children":1382},{"class":551,"line":1381},10,[1383,1387,1391,1395],{"type":55,"tag":549,"props":1384,"children":1385},{"style":568},[1386],{"type":60,"value":1208},{"type":55,"tag":549,"props":1388,"children":1389},{"style":908},[1390],{"type":60,"value":911},{"type":55,"tag":549,"props":1392,"children":1393},{"style":568},[1394],{"type":60,"value":916},{"type":55,"tag":549,"props":1396,"children":1397},{"style":562},[1398],{"type":60,"value":1054},{"type":55,"tag":549,"props":1400,"children":1402},{"class":551,"line":1401},11,[1403,1407,1411],{"type":55,"tag":549,"props":1404,"children":1405},{"style":908},[1406],{"type":60,"value":1314},{"type":55,"tag":549,"props":1408,"children":1409},{"style":568},[1410],{"type":60,"value":916},{"type":55,"tag":549,"props":1412,"children":1413},{"style":562},[1414],{"type":60,"value":1415}," output.status === 'approved'\n",{"type":55,"tag":91,"props":1417,"children":1419},{"id":1418},"output-contract",[1420],{"type":60,"value":1421},"Output Contract",{"type":55,"tag":63,"props":1423,"children":1424},{},[1425],{"type":60,"value":1426},"When done, state:",{"type":55,"tag":103,"props":1428,"children":1429},{},[1430,1435,1440,1445,1450,1455,1460],{"type":55,"tag":107,"props":1431,"children":1432},{},[1433],{"type":60,"value":1434},"Eval question and target\u002Fprovider used",{"type":55,"tag":107,"props":1436,"children":1437},{},[1438],{"type":60,"value":1439},"Files created or changed",{"type":55,"tag":107,"props":1441,"children":1442},{},[1443],{"type":60,"value":1444},"Assertion strategy and why",{"type":55,"tag":107,"props":1446,"children":1447},{},[1448],{"type":60,"value":1449},"Validation\u002Feval commands run",{"type":55,"tag":107,"props":1451,"children":1452},{},[1453],{"type":60,"value":1454},"Result stats and any failures\u002Ferrors",{"type":55,"tag":107,"props":1456,"children":1457},{},[1458],{"type":60,"value":1459},"Required environment variables",{"type":55,"tag":107,"props":1461,"children":1462},{},[1463],{"type":60,"value":1464},"Follow-up cases or assertions to add next",{"type":55,"tag":1466,"props":1467,"children":1468},"style",{},[1469],{"type":60,"value":1470},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1472,"total":941},[1473,1480,1492,1507],{"slug":4,"name":4,"fn":5,"description":6,"org":1474,"tags":1475,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1476,1477,1478,1479],{"name":17,"slug":18,"type":12},{"name":14,"slug":15,"type":12},{"name":8,"slug":8,"type":12},{"name":20,"slug":21,"type":12},{"slug":122,"name":122,"fn":1481,"description":1482,"org":1483,"tags":1484,"stars":22,"repoUrl":23,"updatedAt":1491},"configure promptfoo providers and targets","Configure promptfoo providers or redteam targets for hosted models, live HTTP APIs, Python\u002FJavaScript local scripts, agent SDKs, or multi-input systems. Use when connecting promptfoo to the system under test, mapping vars, auth env vars, request bodies, response transforms, or static-code-derived provider wrappers. Do not use for choosing eval assertions or red team plugins unless a smoke test is needed to verify the connection.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1485,1488,1489,1490],{"name":1486,"slug":1487,"type":12},"API Development","api-development",{"name":17,"slug":18,"type":12},{"name":14,"slug":15,"type":12},{"name":20,"slug":21,"type":12},"2026-07-18T05:47:13.657873",{"slug":170,"name":170,"fn":1493,"description":1494,"org":1495,"tags":1496,"stars":22,"repoUrl":23,"updatedAt":1506},"run promptfoo redteam security scans","Run, rerun, inspect, and QA promptfoo redteam scans from generated redteam YAML or an existing redteam setup config. Use when executing `promptfoo redteam eval` or `promptfoo redteam run`, exporting results, triaging attack success rate, grader failures, target errors, filter\u002Frerun commands, reports, or CI gates. Do not use for initial provider wiring or for choosing plugins and strategies before generation.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1497,1500,1501,1502,1505],{"name":1498,"slug":1499,"type":12},"Code Analysis","code-analysis",{"name":17,"slug":18,"type":12},{"name":8,"slug":8,"type":12},{"name":1503,"slug":1504,"type":12},"Security","security",{"name":20,"slug":21,"type":12},"2026-07-18T05:48:03.576002",{"slug":162,"name":162,"fn":1508,"description":1509,"org":1510,"tags":1511,"stars":22,"repoUrl":23,"updatedAt":1518},"configure promptfoo redteam security setups","Create or refine promptfoo redteam setup configs: purpose, targets, plugins, strategies, frameworks, multi-input target inputs, policy text, grader guidance, contexts, and static-code-derived target\u002Fthreat mapping. Use when preparing a red team scan plan from live probes, code evidence, or provider configs, or when generating adversarial test cases for QA. Do not use for basic provider wiring alone or for running\u002Fevaluating an already-generated redteam scan.\n",{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1512,1515,1516,1517],{"name":1513,"slug":1514,"type":12},"Configuration","configuration",{"name":17,"slug":18,"type":12},{"name":8,"slug":8,"type":12},{"name":1503,"slug":1504,"type":12},"2026-07-18T05:48:36.82567",{"items":1520,"total":941},[1521,1528,1535,1543],{"slug":4,"name":4,"fn":5,"description":6,"org":1522,"tags":1523,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1524,1525,1526,1527],{"name":17,"slug":18,"type":12},{"name":14,"slug":15,"type":12},{"name":8,"slug":8,"type":12},{"name":20,"slug":21,"type":12},{"slug":122,"name":122,"fn":1481,"description":1482,"org":1529,"tags":1530,"stars":22,"repoUrl":23,"updatedAt":1491},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1531,1532,1533,1534],{"name":1486,"slug":1487,"type":12},{"name":17,"slug":18,"type":12},{"name":14,"slug":15,"type":12},{"name":20,"slug":21,"type":12},{"slug":170,"name":170,"fn":1493,"description":1494,"org":1536,"tags":1537,"stars":22,"repoUrl":23,"updatedAt":1506},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1538,1539,1540,1541,1542],{"name":1498,"slug":1499,"type":12},{"name":17,"slug":18,"type":12},{"name":8,"slug":8,"type":12},{"name":1503,"slug":1504,"type":12},{"name":20,"slug":21,"type":12},{"slug":162,"name":162,"fn":1508,"description":1509,"org":1544,"tags":1545,"stars":22,"repoUrl":23,"updatedAt":1518},{"slug":8,"name":8,"logoUrl":9,"githubOrg":8},[1546,1547,1548,1549],{"name":1513,"slug":1514,"type":12},{"name":17,"slug":18,"type":12},{"name":8,"slug":8,"type":12},{"name":1503,"slug":1504,"type":12}]