[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-launchdarkly-online-evals":3,"mdc--cxtpru-key":34,"related-org-launchdarkly-online-evals":3837,"related-repo-launchdarkly-online-evals":3970},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":29,"sourceUrl":32,"mdContent":33},"online-evals","configure online LLM-as-a-judge evaluations","Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"launchdarkly","LaunchDarkly","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Flaunchdarkly.png",[12,16,19],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},20,"https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fai-tooling","2026-07-28T05:33:34.700151",null,6,[26,27,28],"agent-skills","launchdarkly-ai","managed-by-terraform",{"repoUrl":21,"stars":20,"forks":24,"topics":30,"description":31},[26,27,28],"LaunchDarkly's official AI tooling","https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fai-tooling\u002Ftree\u002FHEAD\u002Fskills\u002Fagentcontrol\u002Fonline-evals","---\nname: online-evals\ndescription: Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.\ncompatibility: Requires LaunchDarkly API access token with ai-configs:write permission. SDK versions Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated `track_judge_result` \u002F `trackJudgeResult` API.\nmetadata:\n  author: launchdarkly\n  version: \"0.1.0\"\n---\n\n# Config Online Evaluations\n\nAttach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.\n\n## Prerequisites\n\n- LaunchDarkly account with AgentControl enabled\n- API access token with write permissions\n- Existing config with variations (use `configs-create` skill)\n- For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+\n\n## API Key Detection\n\n1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY`\n2. **Check MCP config** - Claude: `~\u002F.claude\u002Fconfig.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY`\n3. **Prompt user** - Only if detection fails\n\n## Core Concepts\n\n### What Are Judges?\n\nJudges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results:\n\n```json\n{\n  \"score\": 0.85,\n  \"reasoning\": \"Answered correctly with one minor omission\"\n}\n```\n\n### Built-in Judges\n\nLaunchDarkly provides three pre-configured judges:\n\n| Judge | Metric Key | Measures |\n|-------|-----------|----------|\n| Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is |\n| Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request |\n| Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) |\n\n### Completion Mode Only\n\nJudges can only be attached to **completion mode** configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK.\n\n### Restrictions\n\n- Cannot attach judges to judges (no recursion)\n- Cannot attach multiple judges with the same metric key to a single variation\n- Cannot view\u002Fedit model parameters or tools on judge variations\n\n## Workflow\n\n### Step 1: Create Custom Judges (Optional)\n\nFor domain-specific evaluation, create judge configs:\n\n```bash\n# Create judge config\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"security-judge\",\n    \"name\": \"Security Judge\",\n    \"mode\": \"judge\",\n    \"evaluationMetricKey\": \"security\",\n    \"isInverted\": false\n  }'\n```\n\n> **Note:** Set `isInverted: true` for metrics like toxicity where 0.0 is better.\n\nThen add a variation with the evaluation prompt:\n\n```bash\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Fvariations\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"default\",\n    \"name\": \"Default\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a security auditor. Score from 0.0 to 1.0:\\n- 1.0: No security issues\\n- 0.7-0.9: Minor issues\\n- 0.4-0.6: Moderate issues\\n- 0.1-0.3: Serious vulnerabilities\\n- 0.0: Critical vulnerabilities\\n\\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\"\n      }\n    ],\n    \"modelConfigKey\": \"OpenAI.gpt-4o-mini\",\n    \"model\": {\n      \"parameters\": {\n        \"temperature\": 0.3\n      }\n    }\n  }'\n```\n\n### Step 2: Attach Judges to Variations\n\nUse the variation PATCH endpoint:\n\n```bash\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002F{configKey}\u002Fvariations\u002F{variationKey}\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"judgeConfiguration\": {\n      \"judges\": [\n        {\"judgeConfigKey\": \"security-judge\", \"samplingRate\": 1.0},\n        {\"judgeConfigKey\": \"api-contract-judge\", \"samplingRate\": 0.5}\n      ]\n    }\n  }'\n```\n\n> **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges.\n\n### Step 3: Set Fallthrough on Judges\n\nEach judge config needs its fallthrough set to the enabled variation. Configs default to the \"disabled\" variation (index 0).\n\n> **Note:** `turnTargetingOn` does not work for configs. Use `updateFallthroughVariationOrRollout` instead.\n\n```bash\n# First get the variation ID for \"Default\" from GET targeting response\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Ftargeting\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson; domain-model=launchdarkly.semanticpatch\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"environmentKey\": \"production\",\n    \"instructions\": [{\n      \"kind\": \"updateFallthroughVariationOrRollout\",\n      \"variationId\": \"your-default-variation-uuid\"\n    }]\n  }'\n```\n\n## Python Implementation\n\n```python\nimport requests\nimport os\nfrom typing import Optional\n\nclass AIConfigJudges:\n    \"\"\"Manager for config judge attachments\"\"\"\n\n    def __init__(self, api_token: str, project_key: str):\n        self.api_token = api_token\n        self.project_key = project_key\n        self.base_url = \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\"\n        self.headers = {\n            \"Authorization\": api_token,\n            \"Content-Type\": \"application\u002Fjson\",\n            \"LD-API-Version\": \"beta\"\n        }\n\n    def attach_judges(self, config_key: str, variation_key: str,\n                      judges: list[dict]) -> dict:\n        \"\"\"\n        Attach judges to a variation.\n\n        Args:\n            config_key: config key\n            variation_key: Variation key\n            judges: List of {\"judgeConfigKey\": str, \"samplingRate\": float}\n        \"\"\"\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Fvariations\u002F{variation_key}\"\n\n        response = requests.patch(url, headers=self.headers, json={\n            \"judgeConfiguration\": {\"judges\": judges}\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Attached {len(judges)} judges to {config_key}\u002F{variation_key}\")\n            return response.json()\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return {}\n\n    def create_judge(self, key: str, name: str, metric_key: str,\n                     system_prompt: str, model: str = \"OpenAI.gpt-4o-mini\",\n                     is_inverted: bool = False) -> dict:\n        \"\"\"\n        Create a judge config.\n\n        Args:\n            key: Judge config key\n            name: Display name\n            metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key})\n            system_prompt: Evaluation instructions\n            is_inverted: True if lower scores are better (e.g., toxicity)\n        \"\"\"\n        # Create config\n        config_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\"\n        response = requests.post(config_url, headers=self.headers, json={\n            \"key\": key,\n            \"name\": name,\n            \"mode\": \"judge\",\n            \"evaluationMetricKey\": metric_key,\n            \"isInverted\": is_inverted\n        })\n\n        if response.status_code not in [200, 201]:\n            print(f\"[ERROR] Creating config: {response.text}\")\n            return {}\n\n        # Create variation\n        var_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{key}\u002Fvariations\"\n        response = requests.post(var_url, headers=self.headers, json={\n            \"key\": \"default\",\n            \"name\": \"Default\",\n            \"messages\": [{\"role\": \"system\", \"content\": system_prompt}],\n            \"modelConfigKey\": model,\n            \"model\": {\"parameters\": {\"temperature\": 0.3}}\n        })\n\n        if response.status_code in [200, 201]:\n            print(f\"[OK] Created judge: {key}\")\n            return response.json()\n        print(f\"[ERROR] Creating variation: {response.text}\")\n        return {}\n\n    def set_fallthrough(self, config_key: str, environment: str,\n                        variation_key: str = \"default\") -> bool:\n        \"\"\"\n        Set fallthrough to enable a judge config.\n\n        Note: turnTargetingOn doesn't work for configs. Instead, set the\n        fallthrough from disabled (index 0) to the enabled variation.\n        \"\"\"\n        # Get variation ID\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Ftargeting\"\n        response = requests.get(url, headers=self.headers)\n\n        if response.status_code != 200:\n            print(f\"[ERROR] {response.status_code}: {response.text}\")\n            return False\n\n        targeting = response.json()\n        variation_id = None\n        for var in targeting.get(\"variations\", []):\n            if var.get(\"key\") == variation_key or var.get(\"name\") == variation_key:\n                variation_id = var.get(\"_id\")\n                break\n\n        if not variation_id:\n            print(f\"[ERROR] Variation '{variation_key}' not found\")\n            return False\n\n        # Set fallthrough\n        response = requests.patch(url, headers={\n            **self.headers,\n            \"Content-Type\": \"application\u002Fjson; domain-model=launchdarkly.semanticpatch\"\n        }, json={\n            \"environmentKey\": environment,\n            \"instructions\": [{\n                \"kind\": \"updateFallthroughVariationOrRollout\",\n                \"variationId\": variation_id\n            }]\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Fallthrough set for {config_key}\")\n            return True\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return False\n```\n\n## SDK: Automatic Evaluation\n\nWhen using `create_model()` + `run()`, attached judges evaluate automatically:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AICompletionConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\nai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    default_value = AICompletionConfigDefault(enabled=False)\n\n    # create_model() initializes with judges from Config\n    model = await aiclient.create_model(ai_config_key, context, default_value, {})\n\n    if not model:\n        print(f\"agent configuration not enabled for: {ai_config_key}\")\n        return\n\n    user_input = 'How can LaunchDarkly help me?'\n\n    # run() automatically evaluates with attached judges\n    result = await model.run(user_input)\n    print(\"Response:\", result.content)\n\n    # Await evaluation results\n    if result.evaluations and len(result.evaluations) > 0:\n        eval_results = await asyncio.gather(*result.evaluations)\n        results_to_display = [\n            r.to_dict() if r is not None else \"not evaluated\"\n            for r in eval_results\n        ]\n        print(\"Judge results:\")\n        print(json.dumps(results_to_display, indent=2, default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n## SDK: Direct Judge Evaluation\n\nFor agent mode or custom pipelines, evaluate input\u002Foutput pairs directly:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AIJudgeConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\njudge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    judge_default_value = AIJudgeConfigDefault(enabled=False)\n\n    # Get judge configuration from LaunchDarkly\n    judge = aiclient.create_judge(judge_key, context, judge_default_value)\n\n    if not judge:\n        print(f\"agent judge configuration not enabled for key: {judge_key}\")\n        return\n\n    input_text = 'You are a helpful assistant. How can you help me?'\n    output_text = 'I can answer any question you have.'\n\n    # Evaluate the input\u002Foutput pair — returns a JudgeResult.\n    judge_result = await judge.evaluate(input_text, output_text)\n\n    if not judge_result.sampled:\n        print(\"Judge evaluation was skipped (sample rate or configuration issue)\")\n        return\n\n    # Track the consolidated result on the Config tracker if needed:\n    # tracker = ai_config.create_tracker()\n    # tracker.track_judge_result(judge_result)\n\n    print(\"Judge Result:\")\n    print(json.dumps(judge_result.to_dict(), default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n> **Note:** Direct evaluation does not automatically record metrics. Obtain a tracker via `ai_config.create_tracker()` \u002F `aiConfig.createTracker()` and call `tracker.track_judge_result(result)` \u002F `tracker.trackJudgeResult(result)` to record scores for the config you're evaluating.\n\n## Sampling Rates\n\nEach evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.\n\nYou can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.\n\n## Viewing Results\n\n1. Navigate to **configs** > select your config\n2. Click **Monitoring** tab\n3. Select **Evaluator metrics** from dropdown\n4. View scores by variation and time range\n\nResults appear within 1-2 minutes of evaluation.\n\n## Use in Guardrails and Experiments\n\nEvaluation metrics integrate with:\n- **Guarded rollouts**: Pause\u002Frevert when scores fall below threshold\n- **Experiments**: Compare variations using evaluation metrics as goals\n\n## Error Handling\n\n| Status | Cause | Solution |\n|--------|-------|----------|\n| 404 | Config\u002Fvariation not found | Verify keys exist |\n| 400 | Invalid judge config | Check judgeConfigKey exists |\n| 403 | Insufficient permissions | Check API token permissions |\n| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |\n\n## Next Steps\n\nAfter attaching judges:\n1. **Set fallthrough** on judge configs to an enabled variation (required)\n2. **Monitor results** in Monitoring tab\n3. **Adjust sampling** based on cost\u002Fcoverage needs\n4. **Set up guarded rollouts** for automatic regression detection\n\n## Related Skills\n\n- `configs-create` - Create configs and judges\n- `configs-targeting` - Configure targeting rules\n- `configs-variations` - Manage variations\n\n## References\n\n- [Online Evaluations](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fonline-evaluations.md)\n- [Custom Judges](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fcustom-judges.md)\n\n**Python SDK examples:**\n- [create_judge_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_judge\u002Fcreate_judge_example.py) - Evaluate input\u002Foutput pairs directly via `create_judge` + `evaluate`\n- [create_model_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_model\u002Fcreate_model_example.py) - Automatic evaluation with `create_model` + `run` (attached judges fire during the run)\n\n**Node.js SDK examples:**\n- [features\u002Fcreate-judge](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-judge\u002Fsrc\u002Findex.ts) - Evaluate input\u002Foutput pairs directly via `createJudge` + `evaluate`\n- [features\u002Fcreate-model](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-model\u002Fsrc\u002Findex.ts) - Automatic evaluation with `createModel` + `run` (attached judges fire during the run)\n",{"data":35,"body":39},{"name":4,"description":6,"compatibility":36,"metadata":37},"Requires LaunchDarkly API access token with ai-configs:write permission. SDK versions Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated `track_judge_result` \u002F `trackJudgeResult` API.",{"author":8,"version":38},"0.1.0",{"type":40,"children":41},"root",[42,51,57,64,98,104,174,180,187,199,304,310,315,414,420,432,438,456,462,468,473,679,701,706,958,964,969,1151,1179,1185,1190,1218,1401,1407,2500,2506,2527,2931,2937,2942,3331,3374,3380,3385,3390,3396,3440,3445,3451,3456,3479,3485,3584,3590,3595,3638,3644,3679,3685,3710,3718,3773,3781,3831],{"type":43,"tag":44,"props":45,"children":47},"element","h1",{"id":46},"config-online-evaluations",[48],{"type":49,"value":50},"text","Config Online Evaluations",{"type":43,"tag":52,"props":53,"children":54},"p",{},[55],{"type":49,"value":56},"Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.",{"type":43,"tag":58,"props":59,"children":61},"h2",{"id":60},"prerequisites",[62],{"type":49,"value":63},"Prerequisites",{"type":43,"tag":65,"props":66,"children":67},"ul",{},[68,74,79,93],{"type":43,"tag":69,"props":70,"children":71},"li",{},[72],{"type":49,"value":73},"LaunchDarkly account with AgentControl enabled",{"type":43,"tag":69,"props":75,"children":76},{},[77],{"type":49,"value":78},"API access token with write permissions",{"type":43,"tag":69,"props":80,"children":81},{},[82,84,91],{"type":49,"value":83},"Existing config with variations (use ",{"type":43,"tag":85,"props":86,"children":88},"code",{"className":87},[],[89],{"type":49,"value":90},"configs-create",{"type":49,"value":92}," skill)",{"type":43,"tag":69,"props":94,"children":95},{},[96],{"type":49,"value":97},"For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+",{"type":43,"tag":58,"props":99,"children":101},{"id":100},"api-key-detection",[102],{"type":49,"value":103},"API Key Detection",{"type":43,"tag":105,"props":106,"children":107},"ol",{},[108,140,164],{"type":43,"tag":69,"props":109,"children":110},{},[111,117,119,125,127,133,134],{"type":43,"tag":112,"props":113,"children":114},"strong",{},[115],{"type":49,"value":116},"Check environment variables",{"type":49,"value":118}," - ",{"type":43,"tag":85,"props":120,"children":122},{"className":121},[],[123],{"type":49,"value":124},"LAUNCHDARKLY_API_KEY",{"type":49,"value":126},", ",{"type":43,"tag":85,"props":128,"children":130},{"className":129},[],[131],{"type":49,"value":132},"LAUNCHDARKLY_API_TOKEN",{"type":49,"value":126},{"type":43,"tag":85,"props":135,"children":137},{"className":136},[],[138],{"type":49,"value":139},"LD_API_KEY",{"type":43,"tag":69,"props":141,"children":142},{},[143,148,150,156,158],{"type":43,"tag":112,"props":144,"children":145},{},[146],{"type":49,"value":147},"Check MCP config",{"type":49,"value":149}," - Claude: ",{"type":43,"tag":85,"props":151,"children":153},{"className":152},[],[154],{"type":49,"value":155},"~\u002F.claude\u002Fconfig.json",{"type":49,"value":157}," -> ",{"type":43,"tag":85,"props":159,"children":161},{"className":160},[],[162],{"type":49,"value":163},"mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY",{"type":43,"tag":69,"props":165,"children":166},{},[167,172],{"type":43,"tag":112,"props":168,"children":169},{},[170],{"type":49,"value":171},"Prompt user",{"type":49,"value":173}," - Only if detection fails",{"type":43,"tag":58,"props":175,"children":177},{"id":176},"core-concepts",[178],{"type":49,"value":179},"Core Concepts",{"type":43,"tag":181,"props":182,"children":184},"h3",{"id":183},"what-are-judges",[185],{"type":49,"value":186},"What Are Judges?",{"type":43,"tag":52,"props":188,"children":189},{},[190,192,197],{"type":49,"value":191},"Judges are specialized configs in ",{"type":43,"tag":112,"props":193,"children":194},{},[195],{"type":49,"value":196},"judge mode",{"type":49,"value":198}," that evaluate responses from other configs. They use an LLM to score outputs and return structured results:",{"type":43,"tag":200,"props":201,"children":206},"pre",{"className":202,"code":203,"language":204,"meta":205,"style":205},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"score\": 0.85,\n  \"reasoning\": \"Answered correctly with one minor omission\"\n}\n","json","",[207],{"type":43,"tag":85,"props":208,"children":209},{"__ignoreMap":205},[210,222,258,295],{"type":43,"tag":211,"props":212,"children":215},"span",{"class":213,"line":214},"line",1,[216],{"type":43,"tag":211,"props":217,"children":219},{"style":218},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[220],{"type":49,"value":221},"{\n",{"type":43,"tag":211,"props":223,"children":225},{"class":213,"line":224},2,[226,231,237,242,247,253],{"type":43,"tag":211,"props":227,"children":228},{"style":218},[229],{"type":49,"value":230},"  \"",{"type":43,"tag":211,"props":232,"children":234},{"style":233},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[235],{"type":49,"value":236},"score",{"type":43,"tag":211,"props":238,"children":239},{"style":218},[240],{"type":49,"value":241},"\"",{"type":43,"tag":211,"props":243,"children":244},{"style":218},[245],{"type":49,"value":246},":",{"type":43,"tag":211,"props":248,"children":250},{"style":249},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[251],{"type":49,"value":252}," 0.85",{"type":43,"tag":211,"props":254,"children":255},{"style":218},[256],{"type":49,"value":257},",\n",{"type":43,"tag":211,"props":259,"children":261},{"class":213,"line":260},3,[262,266,271,275,279,284,290],{"type":43,"tag":211,"props":263,"children":264},{"style":218},[265],{"type":49,"value":230},{"type":43,"tag":211,"props":267,"children":268},{"style":233},[269],{"type":49,"value":270},"reasoning",{"type":43,"tag":211,"props":272,"children":273},{"style":218},[274],{"type":49,"value":241},{"type":43,"tag":211,"props":276,"children":277},{"style":218},[278],{"type":49,"value":246},{"type":43,"tag":211,"props":280,"children":281},{"style":218},[282],{"type":49,"value":283}," \"",{"type":43,"tag":211,"props":285,"children":287},{"style":286},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[288],{"type":49,"value":289},"Answered correctly with one minor omission",{"type":43,"tag":211,"props":291,"children":292},{"style":218},[293],{"type":49,"value":294},"\"\n",{"type":43,"tag":211,"props":296,"children":298},{"class":213,"line":297},4,[299],{"type":43,"tag":211,"props":300,"children":301},{"style":218},[302],{"type":49,"value":303},"}\n",{"type":43,"tag":181,"props":305,"children":307},{"id":306},"built-in-judges",[308],{"type":49,"value":309},"Built-in Judges",{"type":43,"tag":52,"props":311,"children":312},{},[313],{"type":49,"value":314},"LaunchDarkly provides three pre-configured judges:",{"type":43,"tag":316,"props":317,"children":318},"table",{},[319,343],{"type":43,"tag":320,"props":321,"children":322},"thead",{},[323],{"type":43,"tag":324,"props":325,"children":326},"tr",{},[327,333,338],{"type":43,"tag":328,"props":329,"children":330},"th",{},[331],{"type":49,"value":332},"Judge",{"type":43,"tag":328,"props":334,"children":335},{},[336],{"type":49,"value":337},"Metric Key",{"type":43,"tag":328,"props":339,"children":340},{},[341],{"type":49,"value":342},"Measures",{"type":43,"tag":344,"props":345,"children":346},"tbody",{},[347,370,392],{"type":43,"tag":324,"props":348,"children":349},{},[350,356,365],{"type":43,"tag":351,"props":352,"children":353},"td",{},[354],{"type":49,"value":355},"Accuracy",{"type":43,"tag":351,"props":357,"children":358},{},[359],{"type":43,"tag":85,"props":360,"children":362},{"className":361},[],[363],{"type":49,"value":364},"$ld:ai:judge:accuracy",{"type":43,"tag":351,"props":366,"children":367},{},[368],{"type":49,"value":369},"How correct and grounded the response is",{"type":43,"tag":324,"props":371,"children":372},{},[373,378,387],{"type":43,"tag":351,"props":374,"children":375},{},[376],{"type":49,"value":377},"Relevance",{"type":43,"tag":351,"props":379,"children":380},{},[381],{"type":43,"tag":85,"props":382,"children":384},{"className":383},[],[385],{"type":49,"value":386},"$ld:ai:judge:relevance",{"type":43,"tag":351,"props":388,"children":389},{},[390],{"type":49,"value":391},"How well it addresses the user request",{"type":43,"tag":324,"props":393,"children":394},{},[395,400,409],{"type":43,"tag":351,"props":396,"children":397},{},[398],{"type":49,"value":399},"Toxicity",{"type":43,"tag":351,"props":401,"children":402},{},[403],{"type":43,"tag":85,"props":404,"children":406},{"className":405},[],[407],{"type":49,"value":408},"$ld:ai:judge:toxicity",{"type":43,"tag":351,"props":410,"children":411},{},[412],{"type":49,"value":413},"Harmful or unsafe phrasing (lower = safer)",{"type":43,"tag":181,"props":415,"children":417},{"id":416},"completion-mode-only",[418],{"type":49,"value":419},"Completion Mode Only",{"type":43,"tag":52,"props":421,"children":422},{},[423,425,430],{"type":49,"value":424},"Judges can only be attached to ",{"type":43,"tag":112,"props":426,"children":427},{},[428],{"type":49,"value":429},"completion mode",{"type":49,"value":431}," configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK.",{"type":43,"tag":181,"props":433,"children":435},{"id":434},"restrictions",[436],{"type":49,"value":437},"Restrictions",{"type":43,"tag":65,"props":439,"children":440},{},[441,446,451],{"type":43,"tag":69,"props":442,"children":443},{},[444],{"type":49,"value":445},"Cannot attach judges to judges (no recursion)",{"type":43,"tag":69,"props":447,"children":448},{},[449],{"type":49,"value":450},"Cannot attach multiple judges with the same metric key to a single variation",{"type":43,"tag":69,"props":452,"children":453},{},[454],{"type":49,"value":455},"Cannot view\u002Fedit model parameters or tools on judge variations",{"type":43,"tag":58,"props":457,"children":459},{"id":458},"workflow",[460],{"type":49,"value":461},"Workflow",{"type":43,"tag":181,"props":463,"children":465},{"id":464},"step-1-create-custom-judges-optional",[466],{"type":49,"value":467},"Step 1: Create Custom Judges (Optional)",{"type":43,"tag":52,"props":469,"children":470},{},[471],{"type":49,"value":472},"For domain-specific evaluation, create judge configs:",{"type":43,"tag":200,"props":474,"children":478},{"className":475,"code":476,"language":477,"meta":205,"style":205},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Create judge config\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"security-judge\",\n    \"name\": \"Security Judge\",\n    \"mode\": \"judge\",\n    \"evaluationMetricKey\": \"security\",\n    \"isInverted\": false\n  }'\n","bash",[479],{"type":43,"tag":85,"props":480,"children":481},{"__ignoreMap":205},[482,491,529,554,578,603,620,629,638,647,656,665],{"type":43,"tag":211,"props":483,"children":484},{"class":213,"line":214},[485],{"type":43,"tag":211,"props":486,"children":488},{"style":487},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[489],{"type":49,"value":490},"# Create judge config\n",{"type":43,"tag":211,"props":492,"children":493},{"class":213,"line":224},[494,500,505,510,514,519,523],{"type":43,"tag":211,"props":495,"children":497},{"style":496},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[498],{"type":49,"value":499},"curl",{"type":43,"tag":211,"props":501,"children":502},{"style":286},[503],{"type":49,"value":504}," -X",{"type":43,"tag":211,"props":506,"children":507},{"style":286},[508],{"type":49,"value":509}," POST",{"type":43,"tag":211,"props":511,"children":512},{"style":218},[513],{"type":49,"value":283},{"type":43,"tag":211,"props":515,"children":516},{"style":286},[517],{"type":49,"value":518},"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs",{"type":43,"tag":211,"props":520,"children":521},{"style":218},[522],{"type":49,"value":241},{"type":43,"tag":211,"props":524,"children":526},{"style":525},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[527],{"type":49,"value":528}," \\\n",{"type":43,"tag":211,"props":530,"children":531},{"class":213,"line":260},[532,537,541,546,550],{"type":43,"tag":211,"props":533,"children":534},{"style":286},[535],{"type":49,"value":536},"  -H",{"type":43,"tag":211,"props":538,"children":539},{"style":218},[540],{"type":49,"value":283},{"type":43,"tag":211,"props":542,"children":543},{"style":286},[544],{"type":49,"value":545},"Authorization: {api_token}",{"type":43,"tag":211,"props":547,"children":548},{"style":218},[549],{"type":49,"value":241},{"type":43,"tag":211,"props":551,"children":552},{"style":525},[553],{"type":49,"value":528},{"type":43,"tag":211,"props":555,"children":556},{"class":213,"line":297},[557,561,565,570,574],{"type":43,"tag":211,"props":558,"children":559},{"style":286},[560],{"type":49,"value":536},{"type":43,"tag":211,"props":562,"children":563},{"style":218},[564],{"type":49,"value":283},{"type":43,"tag":211,"props":566,"children":567},{"style":286},[568],{"type":49,"value":569},"Content-Type: application\u002Fjson",{"type":43,"tag":211,"props":571,"children":572},{"style":218},[573],{"type":49,"value":241},{"type":43,"tag":211,"props":575,"children":576},{"style":525},[577],{"type":49,"value":528},{"type":43,"tag":211,"props":579,"children":581},{"class":213,"line":580},5,[582,586,590,595,599],{"type":43,"tag":211,"props":583,"children":584},{"style":286},[585],{"type":49,"value":536},{"type":43,"tag":211,"props":587,"children":588},{"style":218},[589],{"type":49,"value":283},{"type":43,"tag":211,"props":591,"children":592},{"style":286},[593],{"type":49,"value":594},"LD-API-Version: beta",{"type":43,"tag":211,"props":596,"children":597},{"style":218},[598],{"type":49,"value":241},{"type":43,"tag":211,"props":600,"children":601},{"style":525},[602],{"type":49,"value":528},{"type":43,"tag":211,"props":604,"children":605},{"class":213,"line":24},[606,611,616],{"type":43,"tag":211,"props":607,"children":608},{"style":286},[609],{"type":49,"value":610},"  -d",{"type":43,"tag":211,"props":612,"children":613},{"style":218},[614],{"type":49,"value":615}," '",{"type":43,"tag":211,"props":617,"children":618},{"style":286},[619],{"type":49,"value":221},{"type":43,"tag":211,"props":621,"children":623},{"class":213,"line":622},7,[624],{"type":43,"tag":211,"props":625,"children":626},{"style":286},[627],{"type":49,"value":628},"    \"key\": \"security-judge\",\n",{"type":43,"tag":211,"props":630,"children":632},{"class":213,"line":631},8,[633],{"type":43,"tag":211,"props":634,"children":635},{"style":286},[636],{"type":49,"value":637},"    \"name\": \"Security Judge\",\n",{"type":43,"tag":211,"props":639,"children":641},{"class":213,"line":640},9,[642],{"type":43,"tag":211,"props":643,"children":644},{"style":286},[645],{"type":49,"value":646},"    \"mode\": \"judge\",\n",{"type":43,"tag":211,"props":648,"children":650},{"class":213,"line":649},10,[651],{"type":43,"tag":211,"props":652,"children":653},{"style":286},[654],{"type":49,"value":655},"    \"evaluationMetricKey\": \"security\",\n",{"type":43,"tag":211,"props":657,"children":659},{"class":213,"line":658},11,[660],{"type":43,"tag":211,"props":661,"children":662},{"style":286},[663],{"type":49,"value":664},"    \"isInverted\": false\n",{"type":43,"tag":211,"props":666,"children":668},{"class":213,"line":667},12,[669,674],{"type":43,"tag":211,"props":670,"children":671},{"style":286},[672],{"type":49,"value":673},"  }",{"type":43,"tag":211,"props":675,"children":676},{"style":218},[677],{"type":49,"value":678},"'\n",{"type":43,"tag":680,"props":681,"children":682},"blockquote",{},[683],{"type":43,"tag":52,"props":684,"children":685},{},[686,691,693,699],{"type":43,"tag":112,"props":687,"children":688},{},[689],{"type":49,"value":690},"Note:",{"type":49,"value":692}," Set ",{"type":43,"tag":85,"props":694,"children":696},{"className":695},[],[697],{"type":49,"value":698},"isInverted: true",{"type":49,"value":700}," for metrics like toxicity where 0.0 is better.",{"type":43,"tag":52,"props":702,"children":703},{},[704],{"type":49,"value":705},"Then add a variation with the evaluation prompt:",{"type":43,"tag":200,"props":707,"children":709},{"className":475,"code":708,"language":477,"meta":205,"style":205},"curl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Fvariations\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"default\",\n    \"name\": \"Default\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a security auditor. Score from 0.0 to 1.0:\\n- 1.0: No security issues\\n- 0.7-0.9: Minor issues\\n- 0.4-0.6: Moderate issues\\n- 0.1-0.3: Serious vulnerabilities\\n- 0.0: Critical vulnerabilities\\n\\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\"\n      }\n    ],\n    \"modelConfigKey\": \"OpenAI.gpt-4o-mini\",\n    \"model\": {\n      \"parameters\": {\n        \"temperature\": 0.3\n      }\n    }\n  }'\n",[710],{"type":43,"tag":85,"props":711,"children":712},{"__ignoreMap":205},[713,745,768,791,814,829,837,845,853,861,869,877,885,894,903,912,921,930,938,947],{"type":43,"tag":211,"props":714,"children":715},{"class":213,"line":214},[716,720,724,728,732,737,741],{"type":43,"tag":211,"props":717,"children":718},{"style":496},[719],{"type":49,"value":499},{"type":43,"tag":211,"props":721,"children":722},{"style":286},[723],{"type":49,"value":504},{"type":43,"tag":211,"props":725,"children":726},{"style":286},[727],{"type":49,"value":509},{"type":43,"tag":211,"props":729,"children":730},{"style":218},[731],{"type":49,"value":283},{"type":43,"tag":211,"props":733,"children":734},{"style":286},[735],{"type":49,"value":736},"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Fvariations",{"type":43,"tag":211,"props":738,"children":739},{"style":218},[740],{"type":49,"value":241},{"type":43,"tag":211,"props":742,"children":743},{"style":525},[744],{"type":49,"value":528},{"type":43,"tag":211,"props":746,"children":747},{"class":213,"line":224},[748,752,756,760,764],{"type":43,"tag":211,"props":749,"children":750},{"style":286},[751],{"type":49,"value":536},{"type":43,"tag":211,"props":753,"children":754},{"style":218},[755],{"type":49,"value":283},{"type":43,"tag":211,"props":757,"children":758},{"style":286},[759],{"type":49,"value":545},{"type":43,"tag":211,"props":761,"children":762},{"style":218},[763],{"type":49,"value":241},{"type":43,"tag":211,"props":765,"children":766},{"style":525},[767],{"type":49,"value":528},{"type":43,"tag":211,"props":769,"children":770},{"class":213,"line":260},[771,775,779,783,787],{"type":43,"tag":211,"props":772,"children":773},{"style":286},[774],{"type":49,"value":536},{"type":43,"tag":211,"props":776,"children":777},{"style":218},[778],{"type":49,"value":283},{"type":43,"tag":211,"props":780,"children":781},{"style":286},[782],{"type":49,"value":569},{"type":43,"tag":211,"props":784,"children":785},{"style":218},[786],{"type":49,"value":241},{"type":43,"tag":211,"props":788,"children":789},{"style":525},[790],{"type":49,"value":528},{"type":43,"tag":211,"props":792,"children":793},{"class":213,"line":297},[794,798,802,806,810],{"type":43,"tag":211,"props":795,"children":796},{"style":286},[797],{"type":49,"value":536},{"type":43,"tag":211,"props":799,"children":800},{"style":218},[801],{"type":49,"value":283},{"type":43,"tag":211,"props":803,"children":804},{"style":286},[805],{"type":49,"value":594},{"type":43,"tag":211,"props":807,"children":808},{"style":218},[809],{"type":49,"value":241},{"type":43,"tag":211,"props":811,"children":812},{"style":525},[813],{"type":49,"value":528},{"type":43,"tag":211,"props":815,"children":816},{"class":213,"line":580},[817,821,825],{"type":43,"tag":211,"props":818,"children":819},{"style":286},[820],{"type":49,"value":610},{"type":43,"tag":211,"props":822,"children":823},{"style":218},[824],{"type":49,"value":615},{"type":43,"tag":211,"props":826,"children":827},{"style":286},[828],{"type":49,"value":221},{"type":43,"tag":211,"props":830,"children":831},{"class":213,"line":24},[832],{"type":43,"tag":211,"props":833,"children":834},{"style":286},[835],{"type":49,"value":836},"    \"key\": \"default\",\n",{"type":43,"tag":211,"props":838,"children":839},{"class":213,"line":622},[840],{"type":43,"tag":211,"props":841,"children":842},{"style":286},[843],{"type":49,"value":844},"    \"name\": \"Default\",\n",{"type":43,"tag":211,"props":846,"children":847},{"class":213,"line":631},[848],{"type":43,"tag":211,"props":849,"children":850},{"style":286},[851],{"type":49,"value":852},"    \"messages\": [\n",{"type":43,"tag":211,"props":854,"children":855},{"class":213,"line":640},[856],{"type":43,"tag":211,"props":857,"children":858},{"style":286},[859],{"type":49,"value":860},"      {\n",{"type":43,"tag":211,"props":862,"children":863},{"class":213,"line":649},[864],{"type":43,"tag":211,"props":865,"children":866},{"style":286},[867],{"type":49,"value":868},"        \"role\": \"system\",\n",{"type":43,"tag":211,"props":870,"children":871},{"class":213,"line":658},[872],{"type":43,"tag":211,"props":873,"children":874},{"style":286},[875],{"type":49,"value":876},"        \"content\": \"You are a security auditor. Score from 0.0 to 1.0:\\n- 1.0: No security issues\\n- 0.7-0.9: Minor issues\\n- 0.4-0.6: Moderate issues\\n- 0.1-0.3: Serious vulnerabilities\\n- 0.0: Critical vulnerabilities\\n\\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\"\n",{"type":43,"tag":211,"props":878,"children":879},{"class":213,"line":667},[880],{"type":43,"tag":211,"props":881,"children":882},{"style":286},[883],{"type":49,"value":884},"      }\n",{"type":43,"tag":211,"props":886,"children":888},{"class":213,"line":887},13,[889],{"type":43,"tag":211,"props":890,"children":891},{"style":286},[892],{"type":49,"value":893},"    ],\n",{"type":43,"tag":211,"props":895,"children":897},{"class":213,"line":896},14,[898],{"type":43,"tag":211,"props":899,"children":900},{"style":286},[901],{"type":49,"value":902},"    \"modelConfigKey\": \"OpenAI.gpt-4o-mini\",\n",{"type":43,"tag":211,"props":904,"children":906},{"class":213,"line":905},15,[907],{"type":43,"tag":211,"props":908,"children":909},{"style":286},[910],{"type":49,"value":911},"    \"model\": {\n",{"type":43,"tag":211,"props":913,"children":915},{"class":213,"line":914},16,[916],{"type":43,"tag":211,"props":917,"children":918},{"style":286},[919],{"type":49,"value":920},"      \"parameters\": {\n",{"type":43,"tag":211,"props":922,"children":924},{"class":213,"line":923},17,[925],{"type":43,"tag":211,"props":926,"children":927},{"style":286},[928],{"type":49,"value":929},"        \"temperature\": 0.3\n",{"type":43,"tag":211,"props":931,"children":933},{"class":213,"line":932},18,[934],{"type":43,"tag":211,"props":935,"children":936},{"style":286},[937],{"type":49,"value":884},{"type":43,"tag":211,"props":939,"children":941},{"class":213,"line":940},19,[942],{"type":43,"tag":211,"props":943,"children":944},{"style":286},[945],{"type":49,"value":946},"    }\n",{"type":43,"tag":211,"props":948,"children":949},{"class":213,"line":20},[950,954],{"type":43,"tag":211,"props":951,"children":952},{"style":286},[953],{"type":49,"value":673},{"type":43,"tag":211,"props":955,"children":956},{"style":218},[957],{"type":49,"value":678},{"type":43,"tag":181,"props":959,"children":961},{"id":960},"step-2-attach-judges-to-variations",[962],{"type":49,"value":963},"Step 2: Attach Judges to Variations",{"type":43,"tag":52,"props":965,"children":966},{},[967],{"type":49,"value":968},"Use the variation PATCH endpoint:",{"type":43,"tag":200,"props":970,"children":972},{"className":475,"code":971,"language":477,"meta":205,"style":205},"curl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002F{configKey}\u002Fvariations\u002F{variationKey}\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"judgeConfiguration\": {\n      \"judges\": [\n        {\"judgeConfigKey\": \"security-judge\", \"samplingRate\": 1.0},\n        {\"judgeConfigKey\": \"api-contract-judge\", \"samplingRate\": 0.5}\n      ]\n    }\n  }'\n",[973],{"type":43,"tag":85,"props":974,"children":975},{"__ignoreMap":205},[976,1009,1032,1055,1078,1093,1101,1109,1117,1125,1133,1140],{"type":43,"tag":211,"props":977,"children":978},{"class":213,"line":214},[979,983,987,992,996,1001,1005],{"type":43,"tag":211,"props":980,"children":981},{"style":496},[982],{"type":49,"value":499},{"type":43,"tag":211,"props":984,"children":985},{"style":286},[986],{"type":49,"value":504},{"type":43,"tag":211,"props":988,"children":989},{"style":286},[990],{"type":49,"value":991}," PATCH",{"type":43,"tag":211,"props":993,"children":994},{"style":218},[995],{"type":49,"value":283},{"type":43,"tag":211,"props":997,"children":998},{"style":286},[999],{"type":49,"value":1000},"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002F{configKey}\u002Fvariations\u002F{variationKey}",{"type":43,"tag":211,"props":1002,"children":1003},{"style":218},[1004],{"type":49,"value":241},{"type":43,"tag":211,"props":1006,"children":1007},{"style":525},[1008],{"type":49,"value":528},{"type":43,"tag":211,"props":1010,"children":1011},{"class":213,"line":224},[1012,1016,1020,1024,1028],{"type":43,"tag":211,"props":1013,"children":1014},{"style":286},[1015],{"type":49,"value":536},{"type":43,"tag":211,"props":1017,"children":1018},{"style":218},[1019],{"type":49,"value":283},{"type":43,"tag":211,"props":1021,"children":1022},{"style":286},[1023],{"type":49,"value":545},{"type":43,"tag":211,"props":1025,"children":1026},{"style":218},[1027],{"type":49,"value":241},{"type":43,"tag":211,"props":1029,"children":1030},{"style":525},[1031],{"type":49,"value":528},{"type":43,"tag":211,"props":1033,"children":1034},{"class":213,"line":260},[1035,1039,1043,1047,1051],{"type":43,"tag":211,"props":1036,"children":1037},{"style":286},[1038],{"type":49,"value":536},{"type":43,"tag":211,"props":1040,"children":1041},{"style":218},[1042],{"type":49,"value":283},{"type":43,"tag":211,"props":1044,"children":1045},{"style":286},[1046],{"type":49,"value":569},{"type":43,"tag":211,"props":1048,"children":1049},{"style":218},[1050],{"type":49,"value":241},{"type":43,"tag":211,"props":1052,"children":1053},{"style":525},[1054],{"type":49,"value":528},{"type":43,"tag":211,"props":1056,"children":1057},{"class":213,"line":297},[1058,1062,1066,1070,1074],{"type":43,"tag":211,"props":1059,"children":1060},{"style":286},[1061],{"type":49,"value":536},{"type":43,"tag":211,"props":1063,"children":1064},{"style":218},[1065],{"type":49,"value":283},{"type":43,"tag":211,"props":1067,"children":1068},{"style":286},[1069],{"type":49,"value":594},{"type":43,"tag":211,"props":1071,"children":1072},{"style":218},[1073],{"type":49,"value":241},{"type":43,"tag":211,"props":1075,"children":1076},{"style":525},[1077],{"type":49,"value":528},{"type":43,"tag":211,"props":1079,"children":1080},{"class":213,"line":580},[1081,1085,1089],{"type":43,"tag":211,"props":1082,"children":1083},{"style":286},[1084],{"type":49,"value":610},{"type":43,"tag":211,"props":1086,"children":1087},{"style":218},[1088],{"type":49,"value":615},{"type":43,"tag":211,"props":1090,"children":1091},{"style":286},[1092],{"type":49,"value":221},{"type":43,"tag":211,"props":1094,"children":1095},{"class":213,"line":24},[1096],{"type":43,"tag":211,"props":1097,"children":1098},{"style":286},[1099],{"type":49,"value":1100},"    \"judgeConfiguration\": {\n",{"type":43,"tag":211,"props":1102,"children":1103},{"class":213,"line":622},[1104],{"type":43,"tag":211,"props":1105,"children":1106},{"style":286},[1107],{"type":49,"value":1108},"      \"judges\": [\n",{"type":43,"tag":211,"props":1110,"children":1111},{"class":213,"line":631},[1112],{"type":43,"tag":211,"props":1113,"children":1114},{"style":286},[1115],{"type":49,"value":1116},"        {\"judgeConfigKey\": \"security-judge\", \"samplingRate\": 1.0},\n",{"type":43,"tag":211,"props":1118,"children":1119},{"class":213,"line":640},[1120],{"type":43,"tag":211,"props":1121,"children":1122},{"style":286},[1123],{"type":49,"value":1124},"        {\"judgeConfigKey\": \"api-contract-judge\", \"samplingRate\": 0.5}\n",{"type":43,"tag":211,"props":1126,"children":1127},{"class":213,"line":649},[1128],{"type":43,"tag":211,"props":1129,"children":1130},{"style":286},[1131],{"type":49,"value":1132},"      ]\n",{"type":43,"tag":211,"props":1134,"children":1135},{"class":213,"line":658},[1136],{"type":43,"tag":211,"props":1137,"children":1138},{"style":286},[1139],{"type":49,"value":946},{"type":43,"tag":211,"props":1141,"children":1142},{"class":213,"line":667},[1143,1147],{"type":43,"tag":211,"props":1144,"children":1145},{"style":286},[1146],{"type":49,"value":673},{"type":43,"tag":211,"props":1148,"children":1149},{"style":218},[1150],{"type":49,"value":678},{"type":43,"tag":680,"props":1152,"children":1153},{},[1154],{"type":43,"tag":52,"props":1155,"children":1156},{},[1157,1162,1164,1170,1172,1177],{"type":43,"tag":112,"props":1158,"children":1159},{},[1160],{"type":49,"value":1161},"Important:",{"type":49,"value":1163}," The ",{"type":43,"tag":85,"props":1165,"children":1167},{"className":1166},[],[1168],{"type":49,"value":1169},"judges",{"type":49,"value":1171}," array ",{"type":43,"tag":112,"props":1173,"children":1174},{},[1175],{"type":49,"value":1176},"replaces all existing",{"type":49,"value":1178}," judge attachments. An empty array removes all judges.",{"type":43,"tag":181,"props":1180,"children":1182},{"id":1181},"step-3-set-fallthrough-on-judges",[1183],{"type":49,"value":1184},"Step 3: Set Fallthrough on Judges",{"type":43,"tag":52,"props":1186,"children":1187},{},[1188],{"type":49,"value":1189},"Each judge config needs its fallthrough set to the enabled variation. Configs default to the \"disabled\" variation (index 0).",{"type":43,"tag":680,"props":1191,"children":1192},{},[1193],{"type":43,"tag":52,"props":1194,"children":1195},{},[1196,1200,1202,1208,1210,1216],{"type":43,"tag":112,"props":1197,"children":1198},{},[1199],{"type":49,"value":690},{"type":49,"value":1201}," ",{"type":43,"tag":85,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":49,"value":1207},"turnTargetingOn",{"type":49,"value":1209}," does not work for configs. Use ",{"type":43,"tag":85,"props":1211,"children":1213},{"className":1212},[],[1214],{"type":49,"value":1215},"updateFallthroughVariationOrRollout",{"type":49,"value":1217}," instead.",{"type":43,"tag":200,"props":1219,"children":1221},{"className":475,"code":1220,"language":477,"meta":205,"style":205},"# First get the variation ID for \"Default\" from GET targeting response\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Ftargeting\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson; domain-model=launchdarkly.semanticpatch\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"environmentKey\": \"production\",\n    \"instructions\": [{\n      \"kind\": \"updateFallthroughVariationOrRollout\",\n      \"variationId\": \"your-default-variation-uuid\"\n    }]\n  }'\n",[1222],{"type":43,"tag":85,"props":1223,"children":1224},{"__ignoreMap":205},[1225,1233,1265,1288,1312,1335,1350,1358,1366,1374,1382,1390],{"type":43,"tag":211,"props":1226,"children":1227},{"class":213,"line":214},[1228],{"type":43,"tag":211,"props":1229,"children":1230},{"style":487},[1231],{"type":49,"value":1232},"# First get the variation ID for \"Default\" from GET targeting response\n",{"type":43,"tag":211,"props":1234,"children":1235},{"class":213,"line":224},[1236,1240,1244,1248,1252,1257,1261],{"type":43,"tag":211,"props":1237,"children":1238},{"style":496},[1239],{"type":49,"value":499},{"type":43,"tag":211,"props":1241,"children":1242},{"style":286},[1243],{"type":49,"value":504},{"type":43,"tag":211,"props":1245,"children":1246},{"style":286},[1247],{"type":49,"value":991},{"type":43,"tag":211,"props":1249,"children":1250},{"style":218},[1251],{"type":49,"value":283},{"type":43,"tag":211,"props":1253,"children":1254},{"style":286},[1255],{"type":49,"value":1256},"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Ftargeting",{"type":43,"tag":211,"props":1258,"children":1259},{"style":218},[1260],{"type":49,"value":241},{"type":43,"tag":211,"props":1262,"children":1263},{"style":525},[1264],{"type":49,"value":528},{"type":43,"tag":211,"props":1266,"children":1267},{"class":213,"line":260},[1268,1272,1276,1280,1284],{"type":43,"tag":211,"props":1269,"children":1270},{"style":286},[1271],{"type":49,"value":536},{"type":43,"tag":211,"props":1273,"children":1274},{"style":218},[1275],{"type":49,"value":283},{"type":43,"tag":211,"props":1277,"children":1278},{"style":286},[1279],{"type":49,"value":545},{"type":43,"tag":211,"props":1281,"children":1282},{"style":218},[1283],{"type":49,"value":241},{"type":43,"tag":211,"props":1285,"children":1286},{"style":525},[1287],{"type":49,"value":528},{"type":43,"tag":211,"props":1289,"children":1290},{"class":213,"line":297},[1291,1295,1299,1304,1308],{"type":43,"tag":211,"props":1292,"children":1293},{"style":286},[1294],{"type":49,"value":536},{"type":43,"tag":211,"props":1296,"children":1297},{"style":218},[1298],{"type":49,"value":283},{"type":43,"tag":211,"props":1300,"children":1301},{"style":286},[1302],{"type":49,"value":1303},"Content-Type: application\u002Fjson; domain-model=launchdarkly.semanticpatch",{"type":43,"tag":211,"props":1305,"children":1306},{"style":218},[1307],{"type":49,"value":241},{"type":43,"tag":211,"props":1309,"children":1310},{"style":525},[1311],{"type":49,"value":528},{"type":43,"tag":211,"props":1313,"children":1314},{"class":213,"line":580},[1315,1319,1323,1327,1331],{"type":43,"tag":211,"props":1316,"children":1317},{"style":286},[1318],{"type":49,"value":536},{"type":43,"tag":211,"props":1320,"children":1321},{"style":218},[1322],{"type":49,"value":283},{"type":43,"tag":211,"props":1324,"children":1325},{"style":286},[1326],{"type":49,"value":594},{"type":43,"tag":211,"props":1328,"children":1329},{"style":218},[1330],{"type":49,"value":241},{"type":43,"tag":211,"props":1332,"children":1333},{"style":525},[1334],{"type":49,"value":528},{"type":43,"tag":211,"props":1336,"children":1337},{"class":213,"line":24},[1338,1342,1346],{"type":43,"tag":211,"props":1339,"children":1340},{"style":286},[1341],{"type":49,"value":610},{"type":43,"tag":211,"props":1343,"children":1344},{"style":218},[1345],{"type":49,"value":615},{"type":43,"tag":211,"props":1347,"children":1348},{"style":286},[1349],{"type":49,"value":221},{"type":43,"tag":211,"props":1351,"children":1352},{"class":213,"line":622},[1353],{"type":43,"tag":211,"props":1354,"children":1355},{"style":286},[1356],{"type":49,"value":1357},"    \"environmentKey\": \"production\",\n",{"type":43,"tag":211,"props":1359,"children":1360},{"class":213,"line":631},[1361],{"type":43,"tag":211,"props":1362,"children":1363},{"style":286},[1364],{"type":49,"value":1365},"    \"instructions\": [{\n",{"type":43,"tag":211,"props":1367,"children":1368},{"class":213,"line":640},[1369],{"type":43,"tag":211,"props":1370,"children":1371},{"style":286},[1372],{"type":49,"value":1373},"      \"kind\": \"updateFallthroughVariationOrRollout\",\n",{"type":43,"tag":211,"props":1375,"children":1376},{"class":213,"line":649},[1377],{"type":43,"tag":211,"props":1378,"children":1379},{"style":286},[1380],{"type":49,"value":1381},"      \"variationId\": \"your-default-variation-uuid\"\n",{"type":43,"tag":211,"props":1383,"children":1384},{"class":213,"line":658},[1385],{"type":43,"tag":211,"props":1386,"children":1387},{"style":286},[1388],{"type":49,"value":1389},"    }]\n",{"type":43,"tag":211,"props":1391,"children":1392},{"class":213,"line":667},[1393,1397],{"type":43,"tag":211,"props":1394,"children":1395},{"style":286},[1396],{"type":49,"value":673},{"type":43,"tag":211,"props":1398,"children":1399},{"style":218},[1400],{"type":49,"value":678},{"type":43,"tag":58,"props":1402,"children":1404},{"id":1403},"python-implementation",[1405],{"type":49,"value":1406},"Python Implementation",{"type":43,"tag":200,"props":1408,"children":1412},{"className":1409,"code":1410,"language":1411,"meta":205,"style":205},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import requests\nimport os\nfrom typing import Optional\n\nclass AIConfigJudges:\n    \"\"\"Manager for config judge attachments\"\"\"\n\n    def __init__(self, api_token: str, project_key: str):\n        self.api_token = api_token\n        self.project_key = project_key\n        self.base_url = \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\"\n        self.headers = {\n            \"Authorization\": api_token,\n            \"Content-Type\": \"application\u002Fjson\",\n            \"LD-API-Version\": \"beta\"\n        }\n\n    def attach_judges(self, config_key: str, variation_key: str,\n                      judges: list[dict]) -> dict:\n        \"\"\"\n        Attach judges to a variation.\n\n        Args:\n            config_key: config key\n            variation_key: Variation key\n            judges: List of {\"judgeConfigKey\": str, \"samplingRate\": float}\n        \"\"\"\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Fvariations\u002F{variation_key}\"\n\n        response = requests.patch(url, headers=self.headers, json={\n            \"judgeConfiguration\": {\"judges\": judges}\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Attached {len(judges)} judges to {config_key}\u002F{variation_key}\")\n            return response.json()\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return {}\n\n    def create_judge(self, key: str, name: str, metric_key: str,\n                     system_prompt: str, model: str = \"OpenAI.gpt-4o-mini\",\n                     is_inverted: bool = False) -> dict:\n        \"\"\"\n        Create a judge config.\n\n        Args:\n            key: Judge config key\n            name: Display name\n            metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key})\n            system_prompt: Evaluation instructions\n            is_inverted: True if lower scores are better (e.g., toxicity)\n        \"\"\"\n        # Create config\n        config_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\"\n        response = requests.post(config_url, headers=self.headers, json={\n            \"key\": key,\n            \"name\": name,\n            \"mode\": \"judge\",\n            \"evaluationMetricKey\": metric_key,\n            \"isInverted\": is_inverted\n        })\n\n        if response.status_code not in [200, 201]:\n            print(f\"[ERROR] Creating config: {response.text}\")\n            return {}\n\n        # Create variation\n        var_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{key}\u002Fvariations\"\n        response = requests.post(var_url, headers=self.headers, json={\n            \"key\": \"default\",\n            \"name\": \"Default\",\n            \"messages\": [{\"role\": \"system\", \"content\": system_prompt}],\n            \"modelConfigKey\": model,\n            \"model\": {\"parameters\": {\"temperature\": 0.3}}\n        })\n\n        if response.status_code in [200, 201]:\n            print(f\"[OK] Created judge: {key}\")\n            return response.json()\n        print(f\"[ERROR] Creating variation: {response.text}\")\n        return {}\n\n    def set_fallthrough(self, config_key: str, environment: str,\n                        variation_key: str = \"default\") -> bool:\n        \"\"\"\n        Set fallthrough to enable a judge config.\n\n        Note: turnTargetingOn doesn't work for configs. Instead, set the\n        fallthrough from disabled (index 0) to the enabled variation.\n        \"\"\"\n        # Get variation ID\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Ftargeting\"\n        response = requests.get(url, headers=self.headers)\n\n        if response.status_code != 200:\n            print(f\"[ERROR] {response.status_code}: {response.text}\")\n            return False\n\n        targeting = response.json()\n        variation_id = None\n        for var in targeting.get(\"variations\", []):\n            if var.get(\"key\") == variation_key or var.get(\"name\") == variation_key:\n                variation_id = var.get(\"_id\")\n                break\n\n        if not variation_id:\n            print(f\"[ERROR] Variation '{variation_key}' not found\")\n            return False\n\n        # Set fallthrough\n        response = requests.patch(url, headers={\n            **self.headers,\n            \"Content-Type\": \"application\u002Fjson; domain-model=launchdarkly.semanticpatch\"\n        }, json={\n            \"environmentKey\": environment,\n            \"instructions\": [{\n                \"kind\": \"updateFallthroughVariationOrRollout\",\n                \"variationId\": variation_id\n            }]\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Fallthrough set for {config_key}\")\n            return True\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return False\n","python",[1413],{"type":43,"tag":85,"props":1414,"children":1415},{"__ignoreMap":205},[1416,1424,1432,1440,1449,1457,1465,1472,1480,1488,1496,1504,1512,1520,1528,1536,1544,1551,1559,1567,1575,1584,1592,1601,1610,1619,1628,1636,1645,1653,1662,1671,1680,1688,1697,1706,1715,1724,1733,1741,1750,1759,1768,1776,1785,1793,1801,1810,1819,1828,1837,1846,1854,1863,1872,1881,1890,1899,1908,1917,1926,1934,1942,1951,1960,1969,1977,1986,1995,2004,2013,2022,2031,2040,2049,2057,2065,2074,2083,2091,2100,2108,2116,2125,2134,2142,2151,2159,2168,2177,2185,2194,2203,2212,2220,2229,2238,2247,2255,2264,2273,2282,2291,2300,2309,2317,2326,2335,2343,2351,2360,2369,2378,2387,2396,2405,2414,2423,2432,2441,2449,2457,2465,2474,2483,2491],{"type":43,"tag":211,"props":1417,"children":1418},{"class":213,"line":214},[1419],{"type":43,"tag":211,"props":1420,"children":1421},{},[1422],{"type":49,"value":1423},"import requests\n",{"type":43,"tag":211,"props":1425,"children":1426},{"class":213,"line":224},[1427],{"type":43,"tag":211,"props":1428,"children":1429},{},[1430],{"type":49,"value":1431},"import os\n",{"type":43,"tag":211,"props":1433,"children":1434},{"class":213,"line":260},[1435],{"type":43,"tag":211,"props":1436,"children":1437},{},[1438],{"type":49,"value":1439},"from typing import Optional\n",{"type":43,"tag":211,"props":1441,"children":1442},{"class":213,"line":297},[1443],{"type":43,"tag":211,"props":1444,"children":1446},{"emptyLinePlaceholder":1445},true,[1447],{"type":49,"value":1448},"\n",{"type":43,"tag":211,"props":1450,"children":1451},{"class":213,"line":580},[1452],{"type":43,"tag":211,"props":1453,"children":1454},{},[1455],{"type":49,"value":1456},"class AIConfigJudges:\n",{"type":43,"tag":211,"props":1458,"children":1459},{"class":213,"line":24},[1460],{"type":43,"tag":211,"props":1461,"children":1462},{},[1463],{"type":49,"value":1464},"    \"\"\"Manager for config judge attachments\"\"\"\n",{"type":43,"tag":211,"props":1466,"children":1467},{"class":213,"line":622},[1468],{"type":43,"tag":211,"props":1469,"children":1470},{"emptyLinePlaceholder":1445},[1471],{"type":49,"value":1448},{"type":43,"tag":211,"props":1473,"children":1474},{"class":213,"line":631},[1475],{"type":43,"tag":211,"props":1476,"children":1477},{},[1478],{"type":49,"value":1479},"    def __init__(self, api_token: str, project_key: str):\n",{"type":43,"tag":211,"props":1481,"children":1482},{"class":213,"line":640},[1483],{"type":43,"tag":211,"props":1484,"children":1485},{},[1486],{"type":49,"value":1487},"        self.api_token = api_token\n",{"type":43,"tag":211,"props":1489,"children":1490},{"class":213,"line":649},[1491],{"type":43,"tag":211,"props":1492,"children":1493},{},[1494],{"type":49,"value":1495},"        self.project_key = project_key\n",{"type":43,"tag":211,"props":1497,"children":1498},{"class":213,"line":658},[1499],{"type":43,"tag":211,"props":1500,"children":1501},{},[1502],{"type":49,"value":1503},"        self.base_url = \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\"\n",{"type":43,"tag":211,"props":1505,"children":1506},{"class":213,"line":667},[1507],{"type":43,"tag":211,"props":1508,"children":1509},{},[1510],{"type":49,"value":1511},"        self.headers = {\n",{"type":43,"tag":211,"props":1513,"children":1514},{"class":213,"line":887},[1515],{"type":43,"tag":211,"props":1516,"children":1517},{},[1518],{"type":49,"value":1519},"            \"Authorization\": api_token,\n",{"type":43,"tag":211,"props":1521,"children":1522},{"class":213,"line":896},[1523],{"type":43,"tag":211,"props":1524,"children":1525},{},[1526],{"type":49,"value":1527},"            \"Content-Type\": \"application\u002Fjson\",\n",{"type":43,"tag":211,"props":1529,"children":1530},{"class":213,"line":905},[1531],{"type":43,"tag":211,"props":1532,"children":1533},{},[1534],{"type":49,"value":1535},"            \"LD-API-Version\": \"beta\"\n",{"type":43,"tag":211,"props":1537,"children":1538},{"class":213,"line":914},[1539],{"type":43,"tag":211,"props":1540,"children":1541},{},[1542],{"type":49,"value":1543},"        }\n",{"type":43,"tag":211,"props":1545,"children":1546},{"class":213,"line":923},[1547],{"type":43,"tag":211,"props":1548,"children":1549},{"emptyLinePlaceholder":1445},[1550],{"type":49,"value":1448},{"type":43,"tag":211,"props":1552,"children":1553},{"class":213,"line":932},[1554],{"type":43,"tag":211,"props":1555,"children":1556},{},[1557],{"type":49,"value":1558},"    def attach_judges(self, config_key: str, variation_key: str,\n",{"type":43,"tag":211,"props":1560,"children":1561},{"class":213,"line":940},[1562],{"type":43,"tag":211,"props":1563,"children":1564},{},[1565],{"type":49,"value":1566},"                      judges: list[dict]) -> dict:\n",{"type":43,"tag":211,"props":1568,"children":1569},{"class":213,"line":20},[1570],{"type":43,"tag":211,"props":1571,"children":1572},{},[1573],{"type":49,"value":1574},"        \"\"\"\n",{"type":43,"tag":211,"props":1576,"children":1578},{"class":213,"line":1577},21,[1579],{"type":43,"tag":211,"props":1580,"children":1581},{},[1582],{"type":49,"value":1583},"        Attach judges to a variation.\n",{"type":43,"tag":211,"props":1585,"children":1587},{"class":213,"line":1586},22,[1588],{"type":43,"tag":211,"props":1589,"children":1590},{"emptyLinePlaceholder":1445},[1591],{"type":49,"value":1448},{"type":43,"tag":211,"props":1593,"children":1595},{"class":213,"line":1594},23,[1596],{"type":43,"tag":211,"props":1597,"children":1598},{},[1599],{"type":49,"value":1600},"        Args:\n",{"type":43,"tag":211,"props":1602,"children":1604},{"class":213,"line":1603},24,[1605],{"type":43,"tag":211,"props":1606,"children":1607},{},[1608],{"type":49,"value":1609},"            config_key: config key\n",{"type":43,"tag":211,"props":1611,"children":1613},{"class":213,"line":1612},25,[1614],{"type":43,"tag":211,"props":1615,"children":1616},{},[1617],{"type":49,"value":1618},"            variation_key: Variation key\n",{"type":43,"tag":211,"props":1620,"children":1622},{"class":213,"line":1621},26,[1623],{"type":43,"tag":211,"props":1624,"children":1625},{},[1626],{"type":49,"value":1627},"            judges: List of {\"judgeConfigKey\": str, \"samplingRate\": float}\n",{"type":43,"tag":211,"props":1629,"children":1631},{"class":213,"line":1630},27,[1632],{"type":43,"tag":211,"props":1633,"children":1634},{},[1635],{"type":49,"value":1574},{"type":43,"tag":211,"props":1637,"children":1639},{"class":213,"line":1638},28,[1640],{"type":43,"tag":211,"props":1641,"children":1642},{},[1643],{"type":49,"value":1644},"        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Fvariations\u002F{variation_key}\"\n",{"type":43,"tag":211,"props":1646,"children":1648},{"class":213,"line":1647},29,[1649],{"type":43,"tag":211,"props":1650,"children":1651},{"emptyLinePlaceholder":1445},[1652],{"type":49,"value":1448},{"type":43,"tag":211,"props":1654,"children":1656},{"class":213,"line":1655},30,[1657],{"type":43,"tag":211,"props":1658,"children":1659},{},[1660],{"type":49,"value":1661},"        response = requests.patch(url, headers=self.headers, json={\n",{"type":43,"tag":211,"props":1663,"children":1665},{"class":213,"line":1664},31,[1666],{"type":43,"tag":211,"props":1667,"children":1668},{},[1669],{"type":49,"value":1670},"            \"judgeConfiguration\": {\"judges\": judges}\n",{"type":43,"tag":211,"props":1672,"children":1674},{"class":213,"line":1673},32,[1675],{"type":43,"tag":211,"props":1676,"children":1677},{},[1678],{"type":49,"value":1679},"        })\n",{"type":43,"tag":211,"props":1681,"children":1683},{"class":213,"line":1682},33,[1684],{"type":43,"tag":211,"props":1685,"children":1686},{"emptyLinePlaceholder":1445},[1687],{"type":49,"value":1448},{"type":43,"tag":211,"props":1689,"children":1691},{"class":213,"line":1690},34,[1692],{"type":43,"tag":211,"props":1693,"children":1694},{},[1695],{"type":49,"value":1696},"        if response.status_code == 200:\n",{"type":43,"tag":211,"props":1698,"children":1700},{"class":213,"line":1699},35,[1701],{"type":43,"tag":211,"props":1702,"children":1703},{},[1704],{"type":49,"value":1705},"            print(f\"[OK] Attached {len(judges)} judges to {config_key}\u002F{variation_key}\")\n",{"type":43,"tag":211,"props":1707,"children":1709},{"class":213,"line":1708},36,[1710],{"type":43,"tag":211,"props":1711,"children":1712},{},[1713],{"type":49,"value":1714},"            return response.json()\n",{"type":43,"tag":211,"props":1716,"children":1718},{"class":213,"line":1717},37,[1719],{"type":43,"tag":211,"props":1720,"children":1721},{},[1722],{"type":49,"value":1723},"        print(f\"[ERROR] {response.status_code}: {response.text}\")\n",{"type":43,"tag":211,"props":1725,"children":1727},{"class":213,"line":1726},38,[1728],{"type":43,"tag":211,"props":1729,"children":1730},{},[1731],{"type":49,"value":1732},"        return {}\n",{"type":43,"tag":211,"props":1734,"children":1736},{"class":213,"line":1735},39,[1737],{"type":43,"tag":211,"props":1738,"children":1739},{"emptyLinePlaceholder":1445},[1740],{"type":49,"value":1448},{"type":43,"tag":211,"props":1742,"children":1744},{"class":213,"line":1743},40,[1745],{"type":43,"tag":211,"props":1746,"children":1747},{},[1748],{"type":49,"value":1749},"    def create_judge(self, key: str, name: str, metric_key: str,\n",{"type":43,"tag":211,"props":1751,"children":1753},{"class":213,"line":1752},41,[1754],{"type":43,"tag":211,"props":1755,"children":1756},{},[1757],{"type":49,"value":1758},"                     system_prompt: str, model: str = \"OpenAI.gpt-4o-mini\",\n",{"type":43,"tag":211,"props":1760,"children":1762},{"class":213,"line":1761},42,[1763],{"type":43,"tag":211,"props":1764,"children":1765},{},[1766],{"type":49,"value":1767},"                     is_inverted: bool = False) -> dict:\n",{"type":43,"tag":211,"props":1769,"children":1771},{"class":213,"line":1770},43,[1772],{"type":43,"tag":211,"props":1773,"children":1774},{},[1775],{"type":49,"value":1574},{"type":43,"tag":211,"props":1777,"children":1779},{"class":213,"line":1778},44,[1780],{"type":43,"tag":211,"props":1781,"children":1782},{},[1783],{"type":49,"value":1784},"        Create a judge config.\n",{"type":43,"tag":211,"props":1786,"children":1788},{"class":213,"line":1787},45,[1789],{"type":43,"tag":211,"props":1790,"children":1791},{"emptyLinePlaceholder":1445},[1792],{"type":49,"value":1448},{"type":43,"tag":211,"props":1794,"children":1796},{"class":213,"line":1795},46,[1797],{"type":43,"tag":211,"props":1798,"children":1799},{},[1800],{"type":49,"value":1600},{"type":43,"tag":211,"props":1802,"children":1804},{"class":213,"line":1803},47,[1805],{"type":43,"tag":211,"props":1806,"children":1807},{},[1808],{"type":49,"value":1809},"            key: Judge config key\n",{"type":43,"tag":211,"props":1811,"children":1813},{"class":213,"line":1812},48,[1814],{"type":43,"tag":211,"props":1815,"children":1816},{},[1817],{"type":49,"value":1818},"            name: Display name\n",{"type":43,"tag":211,"props":1820,"children":1822},{"class":213,"line":1821},49,[1823],{"type":43,"tag":211,"props":1824,"children":1825},{},[1826],{"type":49,"value":1827},"            metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key})\n",{"type":43,"tag":211,"props":1829,"children":1831},{"class":213,"line":1830},50,[1832],{"type":43,"tag":211,"props":1833,"children":1834},{},[1835],{"type":49,"value":1836},"            system_prompt: Evaluation instructions\n",{"type":43,"tag":211,"props":1838,"children":1840},{"class":213,"line":1839},51,[1841],{"type":43,"tag":211,"props":1842,"children":1843},{},[1844],{"type":49,"value":1845},"            is_inverted: True if lower scores are better (e.g., toxicity)\n",{"type":43,"tag":211,"props":1847,"children":1849},{"class":213,"line":1848},52,[1850],{"type":43,"tag":211,"props":1851,"children":1852},{},[1853],{"type":49,"value":1574},{"type":43,"tag":211,"props":1855,"children":1857},{"class":213,"line":1856},53,[1858],{"type":43,"tag":211,"props":1859,"children":1860},{},[1861],{"type":49,"value":1862},"        # Create config\n",{"type":43,"tag":211,"props":1864,"children":1866},{"class":213,"line":1865},54,[1867],{"type":43,"tag":211,"props":1868,"children":1869},{},[1870],{"type":49,"value":1871},"        config_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\"\n",{"type":43,"tag":211,"props":1873,"children":1875},{"class":213,"line":1874},55,[1876],{"type":43,"tag":211,"props":1877,"children":1878},{},[1879],{"type":49,"value":1880},"        response = requests.post(config_url, headers=self.headers, json={\n",{"type":43,"tag":211,"props":1882,"children":1884},{"class":213,"line":1883},56,[1885],{"type":43,"tag":211,"props":1886,"children":1887},{},[1888],{"type":49,"value":1889},"            \"key\": key,\n",{"type":43,"tag":211,"props":1891,"children":1893},{"class":213,"line":1892},57,[1894],{"type":43,"tag":211,"props":1895,"children":1896},{},[1897],{"type":49,"value":1898},"            \"name\": name,\n",{"type":43,"tag":211,"props":1900,"children":1902},{"class":213,"line":1901},58,[1903],{"type":43,"tag":211,"props":1904,"children":1905},{},[1906],{"type":49,"value":1907},"            \"mode\": \"judge\",\n",{"type":43,"tag":211,"props":1909,"children":1911},{"class":213,"line":1910},59,[1912],{"type":43,"tag":211,"props":1913,"children":1914},{},[1915],{"type":49,"value":1916},"            \"evaluationMetricKey\": metric_key,\n",{"type":43,"tag":211,"props":1918,"children":1920},{"class":213,"line":1919},60,[1921],{"type":43,"tag":211,"props":1922,"children":1923},{},[1924],{"type":49,"value":1925},"            \"isInverted\": is_inverted\n",{"type":43,"tag":211,"props":1927,"children":1929},{"class":213,"line":1928},61,[1930],{"type":43,"tag":211,"props":1931,"children":1932},{},[1933],{"type":49,"value":1679},{"type":43,"tag":211,"props":1935,"children":1937},{"class":213,"line":1936},62,[1938],{"type":43,"tag":211,"props":1939,"children":1940},{"emptyLinePlaceholder":1445},[1941],{"type":49,"value":1448},{"type":43,"tag":211,"props":1943,"children":1945},{"class":213,"line":1944},63,[1946],{"type":43,"tag":211,"props":1947,"children":1948},{},[1949],{"type":49,"value":1950},"        if response.status_code not in [200, 201]:\n",{"type":43,"tag":211,"props":1952,"children":1954},{"class":213,"line":1953},64,[1955],{"type":43,"tag":211,"props":1956,"children":1957},{},[1958],{"type":49,"value":1959},"            print(f\"[ERROR] Creating config: {response.text}\")\n",{"type":43,"tag":211,"props":1961,"children":1963},{"class":213,"line":1962},65,[1964],{"type":43,"tag":211,"props":1965,"children":1966},{},[1967],{"type":49,"value":1968},"            return {}\n",{"type":43,"tag":211,"props":1970,"children":1972},{"class":213,"line":1971},66,[1973],{"type":43,"tag":211,"props":1974,"children":1975},{"emptyLinePlaceholder":1445},[1976],{"type":49,"value":1448},{"type":43,"tag":211,"props":1978,"children":1980},{"class":213,"line":1979},67,[1981],{"type":43,"tag":211,"props":1982,"children":1983},{},[1984],{"type":49,"value":1985},"        # Create variation\n",{"type":43,"tag":211,"props":1987,"children":1989},{"class":213,"line":1988},68,[1990],{"type":43,"tag":211,"props":1991,"children":1992},{},[1993],{"type":49,"value":1994},"        var_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{key}\u002Fvariations\"\n",{"type":43,"tag":211,"props":1996,"children":1998},{"class":213,"line":1997},69,[1999],{"type":43,"tag":211,"props":2000,"children":2001},{},[2002],{"type":49,"value":2003},"        response = requests.post(var_url, headers=self.headers, json={\n",{"type":43,"tag":211,"props":2005,"children":2007},{"class":213,"line":2006},70,[2008],{"type":43,"tag":211,"props":2009,"children":2010},{},[2011],{"type":49,"value":2012},"            \"key\": \"default\",\n",{"type":43,"tag":211,"props":2014,"children":2016},{"class":213,"line":2015},71,[2017],{"type":43,"tag":211,"props":2018,"children":2019},{},[2020],{"type":49,"value":2021},"            \"name\": \"Default\",\n",{"type":43,"tag":211,"props":2023,"children":2025},{"class":213,"line":2024},72,[2026],{"type":43,"tag":211,"props":2027,"children":2028},{},[2029],{"type":49,"value":2030},"            \"messages\": [{\"role\": \"system\", \"content\": system_prompt}],\n",{"type":43,"tag":211,"props":2032,"children":2034},{"class":213,"line":2033},73,[2035],{"type":43,"tag":211,"props":2036,"children":2037},{},[2038],{"type":49,"value":2039},"            \"modelConfigKey\": model,\n",{"type":43,"tag":211,"props":2041,"children":2043},{"class":213,"line":2042},74,[2044],{"type":43,"tag":211,"props":2045,"children":2046},{},[2047],{"type":49,"value":2048},"            \"model\": {\"parameters\": {\"temperature\": 0.3}}\n",{"type":43,"tag":211,"props":2050,"children":2052},{"class":213,"line":2051},75,[2053],{"type":43,"tag":211,"props":2054,"children":2055},{},[2056],{"type":49,"value":1679},{"type":43,"tag":211,"props":2058,"children":2060},{"class":213,"line":2059},76,[2061],{"type":43,"tag":211,"props":2062,"children":2063},{"emptyLinePlaceholder":1445},[2064],{"type":49,"value":1448},{"type":43,"tag":211,"props":2066,"children":2068},{"class":213,"line":2067},77,[2069],{"type":43,"tag":211,"props":2070,"children":2071},{},[2072],{"type":49,"value":2073},"        if response.status_code in [200, 201]:\n",{"type":43,"tag":211,"props":2075,"children":2077},{"class":213,"line":2076},78,[2078],{"type":43,"tag":211,"props":2079,"children":2080},{},[2081],{"type":49,"value":2082},"            print(f\"[OK] Created judge: {key}\")\n",{"type":43,"tag":211,"props":2084,"children":2086},{"class":213,"line":2085},79,[2087],{"type":43,"tag":211,"props":2088,"children":2089},{},[2090],{"type":49,"value":1714},{"type":43,"tag":211,"props":2092,"children":2094},{"class":213,"line":2093},80,[2095],{"type":43,"tag":211,"props":2096,"children":2097},{},[2098],{"type":49,"value":2099},"        print(f\"[ERROR] Creating variation: {response.text}\")\n",{"type":43,"tag":211,"props":2101,"children":2103},{"class":213,"line":2102},81,[2104],{"type":43,"tag":211,"props":2105,"children":2106},{},[2107],{"type":49,"value":1732},{"type":43,"tag":211,"props":2109,"children":2111},{"class":213,"line":2110},82,[2112],{"type":43,"tag":211,"props":2113,"children":2114},{"emptyLinePlaceholder":1445},[2115],{"type":49,"value":1448},{"type":43,"tag":211,"props":2117,"children":2119},{"class":213,"line":2118},83,[2120],{"type":43,"tag":211,"props":2121,"children":2122},{},[2123],{"type":49,"value":2124},"    def set_fallthrough(self, config_key: str, environment: str,\n",{"type":43,"tag":211,"props":2126,"children":2128},{"class":213,"line":2127},84,[2129],{"type":43,"tag":211,"props":2130,"children":2131},{},[2132],{"type":49,"value":2133},"                        variation_key: str = \"default\") -> bool:\n",{"type":43,"tag":211,"props":2135,"children":2137},{"class":213,"line":2136},85,[2138],{"type":43,"tag":211,"props":2139,"children":2140},{},[2141],{"type":49,"value":1574},{"type":43,"tag":211,"props":2143,"children":2145},{"class":213,"line":2144},86,[2146],{"type":43,"tag":211,"props":2147,"children":2148},{},[2149],{"type":49,"value":2150},"        Set fallthrough to enable a judge config.\n",{"type":43,"tag":211,"props":2152,"children":2154},{"class":213,"line":2153},87,[2155],{"type":43,"tag":211,"props":2156,"children":2157},{"emptyLinePlaceholder":1445},[2158],{"type":49,"value":1448},{"type":43,"tag":211,"props":2160,"children":2162},{"class":213,"line":2161},88,[2163],{"type":43,"tag":211,"props":2164,"children":2165},{},[2166],{"type":49,"value":2167},"        Note: turnTargetingOn doesn't work for configs. Instead, set the\n",{"type":43,"tag":211,"props":2169,"children":2171},{"class":213,"line":2170},89,[2172],{"type":43,"tag":211,"props":2173,"children":2174},{},[2175],{"type":49,"value":2176},"        fallthrough from disabled (index 0) to the enabled variation.\n",{"type":43,"tag":211,"props":2178,"children":2180},{"class":213,"line":2179},90,[2181],{"type":43,"tag":211,"props":2182,"children":2183},{},[2184],{"type":49,"value":1574},{"type":43,"tag":211,"props":2186,"children":2188},{"class":213,"line":2187},91,[2189],{"type":43,"tag":211,"props":2190,"children":2191},{},[2192],{"type":49,"value":2193},"        # Get variation ID\n",{"type":43,"tag":211,"props":2195,"children":2197},{"class":213,"line":2196},92,[2198],{"type":43,"tag":211,"props":2199,"children":2200},{},[2201],{"type":49,"value":2202},"        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Ftargeting\"\n",{"type":43,"tag":211,"props":2204,"children":2206},{"class":213,"line":2205},93,[2207],{"type":43,"tag":211,"props":2208,"children":2209},{},[2210],{"type":49,"value":2211},"        response = requests.get(url, headers=self.headers)\n",{"type":43,"tag":211,"props":2213,"children":2215},{"class":213,"line":2214},94,[2216],{"type":43,"tag":211,"props":2217,"children":2218},{"emptyLinePlaceholder":1445},[2219],{"type":49,"value":1448},{"type":43,"tag":211,"props":2221,"children":2223},{"class":213,"line":2222},95,[2224],{"type":43,"tag":211,"props":2225,"children":2226},{},[2227],{"type":49,"value":2228},"        if response.status_code != 200:\n",{"type":43,"tag":211,"props":2230,"children":2232},{"class":213,"line":2231},96,[2233],{"type":43,"tag":211,"props":2234,"children":2235},{},[2236],{"type":49,"value":2237},"            print(f\"[ERROR] {response.status_code}: {response.text}\")\n",{"type":43,"tag":211,"props":2239,"children":2241},{"class":213,"line":2240},97,[2242],{"type":43,"tag":211,"props":2243,"children":2244},{},[2245],{"type":49,"value":2246},"            return False\n",{"type":43,"tag":211,"props":2248,"children":2250},{"class":213,"line":2249},98,[2251],{"type":43,"tag":211,"props":2252,"children":2253},{"emptyLinePlaceholder":1445},[2254],{"type":49,"value":1448},{"type":43,"tag":211,"props":2256,"children":2258},{"class":213,"line":2257},99,[2259],{"type":43,"tag":211,"props":2260,"children":2261},{},[2262],{"type":49,"value":2263},"        targeting = response.json()\n",{"type":43,"tag":211,"props":2265,"children":2267},{"class":213,"line":2266},100,[2268],{"type":43,"tag":211,"props":2269,"children":2270},{},[2271],{"type":49,"value":2272},"        variation_id = None\n",{"type":43,"tag":211,"props":2274,"children":2276},{"class":213,"line":2275},101,[2277],{"type":43,"tag":211,"props":2278,"children":2279},{},[2280],{"type":49,"value":2281},"        for var in targeting.get(\"variations\", []):\n",{"type":43,"tag":211,"props":2283,"children":2285},{"class":213,"line":2284},102,[2286],{"type":43,"tag":211,"props":2287,"children":2288},{},[2289],{"type":49,"value":2290},"            if var.get(\"key\") == variation_key or var.get(\"name\") == variation_key:\n",{"type":43,"tag":211,"props":2292,"children":2294},{"class":213,"line":2293},103,[2295],{"type":43,"tag":211,"props":2296,"children":2297},{},[2298],{"type":49,"value":2299},"                variation_id = var.get(\"_id\")\n",{"type":43,"tag":211,"props":2301,"children":2303},{"class":213,"line":2302},104,[2304],{"type":43,"tag":211,"props":2305,"children":2306},{},[2307],{"type":49,"value":2308},"                break\n",{"type":43,"tag":211,"props":2310,"children":2312},{"class":213,"line":2311},105,[2313],{"type":43,"tag":211,"props":2314,"children":2315},{"emptyLinePlaceholder":1445},[2316],{"type":49,"value":1448},{"type":43,"tag":211,"props":2318,"children":2320},{"class":213,"line":2319},106,[2321],{"type":43,"tag":211,"props":2322,"children":2323},{},[2324],{"type":49,"value":2325},"        if not variation_id:\n",{"type":43,"tag":211,"props":2327,"children":2329},{"class":213,"line":2328},107,[2330],{"type":43,"tag":211,"props":2331,"children":2332},{},[2333],{"type":49,"value":2334},"            print(f\"[ERROR] Variation '{variation_key}' not found\")\n",{"type":43,"tag":211,"props":2336,"children":2338},{"class":213,"line":2337},108,[2339],{"type":43,"tag":211,"props":2340,"children":2341},{},[2342],{"type":49,"value":2246},{"type":43,"tag":211,"props":2344,"children":2346},{"class":213,"line":2345},109,[2347],{"type":43,"tag":211,"props":2348,"children":2349},{"emptyLinePlaceholder":1445},[2350],{"type":49,"value":1448},{"type":43,"tag":211,"props":2352,"children":2354},{"class":213,"line":2353},110,[2355],{"type":43,"tag":211,"props":2356,"children":2357},{},[2358],{"type":49,"value":2359},"        # Set fallthrough\n",{"type":43,"tag":211,"props":2361,"children":2363},{"class":213,"line":2362},111,[2364],{"type":43,"tag":211,"props":2365,"children":2366},{},[2367],{"type":49,"value":2368},"        response = requests.patch(url, headers={\n",{"type":43,"tag":211,"props":2370,"children":2372},{"class":213,"line":2371},112,[2373],{"type":43,"tag":211,"props":2374,"children":2375},{},[2376],{"type":49,"value":2377},"            **self.headers,\n",{"type":43,"tag":211,"props":2379,"children":2381},{"class":213,"line":2380},113,[2382],{"type":43,"tag":211,"props":2383,"children":2384},{},[2385],{"type":49,"value":2386},"            \"Content-Type\": \"application\u002Fjson; domain-model=launchdarkly.semanticpatch\"\n",{"type":43,"tag":211,"props":2388,"children":2390},{"class":213,"line":2389},114,[2391],{"type":43,"tag":211,"props":2392,"children":2393},{},[2394],{"type":49,"value":2395},"        }, json={\n",{"type":43,"tag":211,"props":2397,"children":2399},{"class":213,"line":2398},115,[2400],{"type":43,"tag":211,"props":2401,"children":2402},{},[2403],{"type":49,"value":2404},"            \"environmentKey\": environment,\n",{"type":43,"tag":211,"props":2406,"children":2408},{"class":213,"line":2407},116,[2409],{"type":43,"tag":211,"props":2410,"children":2411},{},[2412],{"type":49,"value":2413},"            \"instructions\": [{\n",{"type":43,"tag":211,"props":2415,"children":2417},{"class":213,"line":2416},117,[2418],{"type":43,"tag":211,"props":2419,"children":2420},{},[2421],{"type":49,"value":2422},"                \"kind\": \"updateFallthroughVariationOrRollout\",\n",{"type":43,"tag":211,"props":2424,"children":2426},{"class":213,"line":2425},118,[2427],{"type":43,"tag":211,"props":2428,"children":2429},{},[2430],{"type":49,"value":2431},"                \"variationId\": variation_id\n",{"type":43,"tag":211,"props":2433,"children":2435},{"class":213,"line":2434},119,[2436],{"type":43,"tag":211,"props":2437,"children":2438},{},[2439],{"type":49,"value":2440},"            }]\n",{"type":43,"tag":211,"props":2442,"children":2444},{"class":213,"line":2443},120,[2445],{"type":43,"tag":211,"props":2446,"children":2447},{},[2448],{"type":49,"value":1679},{"type":43,"tag":211,"props":2450,"children":2452},{"class":213,"line":2451},121,[2453],{"type":43,"tag":211,"props":2454,"children":2455},{"emptyLinePlaceholder":1445},[2456],{"type":49,"value":1448},{"type":43,"tag":211,"props":2458,"children":2460},{"class":213,"line":2459},122,[2461],{"type":43,"tag":211,"props":2462,"children":2463},{},[2464],{"type":49,"value":1696},{"type":43,"tag":211,"props":2466,"children":2468},{"class":213,"line":2467},123,[2469],{"type":43,"tag":211,"props":2470,"children":2471},{},[2472],{"type":49,"value":2473},"            print(f\"[OK] Fallthrough set for {config_key}\")\n",{"type":43,"tag":211,"props":2475,"children":2477},{"class":213,"line":2476},124,[2478],{"type":43,"tag":211,"props":2479,"children":2480},{},[2481],{"type":49,"value":2482},"            return True\n",{"type":43,"tag":211,"props":2484,"children":2486},{"class":213,"line":2485},125,[2487],{"type":43,"tag":211,"props":2488,"children":2489},{},[2490],{"type":49,"value":1723},{"type":43,"tag":211,"props":2492,"children":2494},{"class":213,"line":2493},126,[2495],{"type":43,"tag":211,"props":2496,"children":2497},{},[2498],{"type":49,"value":2499},"        return False\n",{"type":43,"tag":58,"props":2501,"children":2503},{"id":2502},"sdk-automatic-evaluation",[2504],{"type":49,"value":2505},"SDK: Automatic Evaluation",{"type":43,"tag":52,"props":2507,"children":2508},{},[2509,2511,2517,2519,2525],{"type":49,"value":2510},"When using ",{"type":43,"tag":85,"props":2512,"children":2514},{"className":2513},[],[2515],{"type":49,"value":2516},"create_model()",{"type":49,"value":2518}," + ",{"type":43,"tag":85,"props":2520,"children":2522},{"className":2521},[],[2523],{"type":49,"value":2524},"run()",{"type":49,"value":2526},", attached judges evaluate automatically:",{"type":43,"tag":200,"props":2528,"children":2530},{"className":1409,"code":2529,"language":1411,"meta":205,"style":205},"import os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AICompletionConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\nai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    default_value = AICompletionConfigDefault(enabled=False)\n\n    # create_model() initializes with judges from Config\n    model = await aiclient.create_model(ai_config_key, context, default_value, {})\n\n    if not model:\n        print(f\"agent configuration not enabled for: {ai_config_key}\")\n        return\n\n    user_input = 'How can LaunchDarkly help me?'\n\n    # run() automatically evaluates with attached judges\n    result = await model.run(user_input)\n    print(\"Response:\", result.content)\n\n    # Await evaluation results\n    if result.evaluations and len(result.evaluations) > 0:\n        eval_results = await asyncio.gather(*result.evaluations)\n        results_to_display = [\n            r.to_dict() if r is not None else \"not evaluated\"\n            for r in eval_results\n        ]\n        print(\"Judge results:\")\n        print(json.dumps(results_to_display, indent=2, default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n",[2531],{"type":43,"tag":85,"props":2532,"children":2533},{"__ignoreMap":205},[2534,2541,2549,2557,2565,2573,2581,2589,2596,2604,2612,2619,2627,2635,2643,2650,2658,2666,2674,2682,2690,2698,2705,2713,2720,2728,2736,2743,2751,2759,2767,2774,2782,2789,2797,2805,2813,2820,2828,2836,2844,2852,2860,2868,2876,2884,2892,2899,2907,2915,2923],{"type":43,"tag":211,"props":2535,"children":2536},{"class":213,"line":214},[2537],{"type":43,"tag":211,"props":2538,"children":2539},{},[2540],{"type":49,"value":1431},{"type":43,"tag":211,"props":2542,"children":2543},{"class":213,"line":224},[2544],{"type":43,"tag":211,"props":2545,"children":2546},{},[2547],{"type":49,"value":2548},"import json\n",{"type":43,"tag":211,"props":2550,"children":2551},{"class":213,"line":260},[2552],{"type":43,"tag":211,"props":2553,"children":2554},{},[2555],{"type":49,"value":2556},"import asyncio\n",{"type":43,"tag":211,"props":2558,"children":2559},{"class":213,"line":297},[2560],{"type":43,"tag":211,"props":2561,"children":2562},{},[2563],{"type":49,"value":2564},"import ldclient\n",{"type":43,"tag":211,"props":2566,"children":2567},{"class":213,"line":580},[2568],{"type":43,"tag":211,"props":2569,"children":2570},{},[2571],{"type":49,"value":2572},"from ldclient import Context\n",{"type":43,"tag":211,"props":2574,"children":2575},{"class":213,"line":24},[2576],{"type":43,"tag":211,"props":2577,"children":2578},{},[2579],{"type":49,"value":2580},"from ldclient.config import Config\n",{"type":43,"tag":211,"props":2582,"children":2583},{"class":213,"line":622},[2584],{"type":43,"tag":211,"props":2585,"children":2586},{},[2587],{"type":49,"value":2588},"from ldai import LDAIClient, AICompletionConfigDefault\n",{"type":43,"tag":211,"props":2590,"children":2591},{"class":213,"line":631},[2592],{"type":43,"tag":211,"props":2593,"children":2594},{"emptyLinePlaceholder":1445},[2595],{"type":49,"value":1448},{"type":43,"tag":211,"props":2597,"children":2598},{"class":213,"line":640},[2599],{"type":43,"tag":211,"props":2600,"children":2601},{},[2602],{"type":49,"value":2603},"sdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\n",{"type":43,"tag":211,"props":2605,"children":2606},{"class":213,"line":649},[2607],{"type":43,"tag":211,"props":2608,"children":2609},{},[2610],{"type":49,"value":2611},"ai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config')\n",{"type":43,"tag":211,"props":2613,"children":2614},{"class":213,"line":658},[2615],{"type":43,"tag":211,"props":2616,"children":2617},{"emptyLinePlaceholder":1445},[2618],{"type":49,"value":1448},{"type":43,"tag":211,"props":2620,"children":2621},{"class":213,"line":667},[2622],{"type":43,"tag":211,"props":2623,"children":2624},{},[2625],{"type":49,"value":2626},"async def async_main():\n",{"type":43,"tag":211,"props":2628,"children":2629},{"class":213,"line":887},[2630],{"type":43,"tag":211,"props":2631,"children":2632},{},[2633],{"type":49,"value":2634},"    ldclient.set_config(Config(sdk_key))\n",{"type":43,"tag":211,"props":2636,"children":2637},{"class":213,"line":896},[2638],{"type":43,"tag":211,"props":2639,"children":2640},{},[2641],{"type":49,"value":2642},"    aiclient = LDAIClient(ldclient.get())\n",{"type":43,"tag":211,"props":2644,"children":2645},{"class":213,"line":905},[2646],{"type":43,"tag":211,"props":2647,"children":2648},{"emptyLinePlaceholder":1445},[2649],{"type":49,"value":1448},{"type":43,"tag":211,"props":2651,"children":2652},{"class":213,"line":914},[2653],{"type":43,"tag":211,"props":2654,"children":2655},{},[2656],{"type":49,"value":2657},"    context = (\n",{"type":43,"tag":211,"props":2659,"children":2660},{"class":213,"line":923},[2661],{"type":43,"tag":211,"props":2662,"children":2663},{},[2664],{"type":49,"value":2665},"        Context.builder('example-user-key')\n",{"type":43,"tag":211,"props":2667,"children":2668},{"class":213,"line":932},[2669],{"type":43,"tag":211,"props":2670,"children":2671},{},[2672],{"type":49,"value":2673},"        .kind('user')\n",{"type":43,"tag":211,"props":2675,"children":2676},{"class":213,"line":940},[2677],{"type":43,"tag":211,"props":2678,"children":2679},{},[2680],{"type":49,"value":2681},"        .name('Sandy')\n",{"type":43,"tag":211,"props":2683,"children":2684},{"class":213,"line":20},[2685],{"type":43,"tag":211,"props":2686,"children":2687},{},[2688],{"type":49,"value":2689},"        .build()\n",{"type":43,"tag":211,"props":2691,"children":2692},{"class":213,"line":1577},[2693],{"type":43,"tag":211,"props":2694,"children":2695},{},[2696],{"type":49,"value":2697},"    )\n",{"type":43,"tag":211,"props":2699,"children":2700},{"class":213,"line":1586},[2701],{"type":43,"tag":211,"props":2702,"children":2703},{"emptyLinePlaceholder":1445},[2704],{"type":49,"value":1448},{"type":43,"tag":211,"props":2706,"children":2707},{"class":213,"line":1594},[2708],{"type":43,"tag":211,"props":2709,"children":2710},{},[2711],{"type":49,"value":2712},"    default_value = AICompletionConfigDefault(enabled=False)\n",{"type":43,"tag":211,"props":2714,"children":2715},{"class":213,"line":1603},[2716],{"type":43,"tag":211,"props":2717,"children":2718},{"emptyLinePlaceholder":1445},[2719],{"type":49,"value":1448},{"type":43,"tag":211,"props":2721,"children":2722},{"class":213,"line":1612},[2723],{"type":43,"tag":211,"props":2724,"children":2725},{},[2726],{"type":49,"value":2727},"    # create_model() initializes with judges from Config\n",{"type":43,"tag":211,"props":2729,"children":2730},{"class":213,"line":1621},[2731],{"type":43,"tag":211,"props":2732,"children":2733},{},[2734],{"type":49,"value":2735},"    model = await aiclient.create_model(ai_config_key, context, default_value, {})\n",{"type":43,"tag":211,"props":2737,"children":2738},{"class":213,"line":1630},[2739],{"type":43,"tag":211,"props":2740,"children":2741},{"emptyLinePlaceholder":1445},[2742],{"type":49,"value":1448},{"type":43,"tag":211,"props":2744,"children":2745},{"class":213,"line":1638},[2746],{"type":43,"tag":211,"props":2747,"children":2748},{},[2749],{"type":49,"value":2750},"    if not model:\n",{"type":43,"tag":211,"props":2752,"children":2753},{"class":213,"line":1647},[2754],{"type":43,"tag":211,"props":2755,"children":2756},{},[2757],{"type":49,"value":2758},"        print(f\"agent configuration not enabled for: {ai_config_key}\")\n",{"type":43,"tag":211,"props":2760,"children":2761},{"class":213,"line":1655},[2762],{"type":43,"tag":211,"props":2763,"children":2764},{},[2765],{"type":49,"value":2766},"        return\n",{"type":43,"tag":211,"props":2768,"children":2769},{"class":213,"line":1664},[2770],{"type":43,"tag":211,"props":2771,"children":2772},{"emptyLinePlaceholder":1445},[2773],{"type":49,"value":1448},{"type":43,"tag":211,"props":2775,"children":2776},{"class":213,"line":1673},[2777],{"type":43,"tag":211,"props":2778,"children":2779},{},[2780],{"type":49,"value":2781},"    user_input = 'How can LaunchDarkly help me?'\n",{"type":43,"tag":211,"props":2783,"children":2784},{"class":213,"line":1682},[2785],{"type":43,"tag":211,"props":2786,"children":2787},{"emptyLinePlaceholder":1445},[2788],{"type":49,"value":1448},{"type":43,"tag":211,"props":2790,"children":2791},{"class":213,"line":1690},[2792],{"type":43,"tag":211,"props":2793,"children":2794},{},[2795],{"type":49,"value":2796},"    # run() automatically evaluates with attached judges\n",{"type":43,"tag":211,"props":2798,"children":2799},{"class":213,"line":1699},[2800],{"type":43,"tag":211,"props":2801,"children":2802},{},[2803],{"type":49,"value":2804},"    result = await model.run(user_input)\n",{"type":43,"tag":211,"props":2806,"children":2807},{"class":213,"line":1708},[2808],{"type":43,"tag":211,"props":2809,"children":2810},{},[2811],{"type":49,"value":2812},"    print(\"Response:\", result.content)\n",{"type":43,"tag":211,"props":2814,"children":2815},{"class":213,"line":1717},[2816],{"type":43,"tag":211,"props":2817,"children":2818},{"emptyLinePlaceholder":1445},[2819],{"type":49,"value":1448},{"type":43,"tag":211,"props":2821,"children":2822},{"class":213,"line":1726},[2823],{"type":43,"tag":211,"props":2824,"children":2825},{},[2826],{"type":49,"value":2827},"    # Await evaluation results\n",{"type":43,"tag":211,"props":2829,"children":2830},{"class":213,"line":1735},[2831],{"type":43,"tag":211,"props":2832,"children":2833},{},[2834],{"type":49,"value":2835},"    if result.evaluations and len(result.evaluations) > 0:\n",{"type":43,"tag":211,"props":2837,"children":2838},{"class":213,"line":1743},[2839],{"type":43,"tag":211,"props":2840,"children":2841},{},[2842],{"type":49,"value":2843},"        eval_results = await asyncio.gather(*result.evaluations)\n",{"type":43,"tag":211,"props":2845,"children":2846},{"class":213,"line":1752},[2847],{"type":43,"tag":211,"props":2848,"children":2849},{},[2850],{"type":49,"value":2851},"        results_to_display = [\n",{"type":43,"tag":211,"props":2853,"children":2854},{"class":213,"line":1761},[2855],{"type":43,"tag":211,"props":2856,"children":2857},{},[2858],{"type":49,"value":2859},"            r.to_dict() if r is not None else \"not evaluated\"\n",{"type":43,"tag":211,"props":2861,"children":2862},{"class":213,"line":1770},[2863],{"type":43,"tag":211,"props":2864,"children":2865},{},[2866],{"type":49,"value":2867},"            for r in eval_results\n",{"type":43,"tag":211,"props":2869,"children":2870},{"class":213,"line":1778},[2871],{"type":43,"tag":211,"props":2872,"children":2873},{},[2874],{"type":49,"value":2875},"        ]\n",{"type":43,"tag":211,"props":2877,"children":2878},{"class":213,"line":1787},[2879],{"type":43,"tag":211,"props":2880,"children":2881},{},[2882],{"type":49,"value":2883},"        print(\"Judge results:\")\n",{"type":43,"tag":211,"props":2885,"children":2886},{"class":213,"line":1795},[2887],{"type":43,"tag":211,"props":2888,"children":2889},{},[2890],{"type":49,"value":2891},"        print(json.dumps(results_to_display, indent=2, default=str))\n",{"type":43,"tag":211,"props":2893,"children":2894},{"class":213,"line":1803},[2895],{"type":43,"tag":211,"props":2896,"children":2897},{"emptyLinePlaceholder":1445},[2898],{"type":49,"value":1448},{"type":43,"tag":211,"props":2900,"children":2901},{"class":213,"line":1812},[2902],{"type":43,"tag":211,"props":2903,"children":2904},{},[2905],{"type":49,"value":2906},"    # Always flush events before closing — trailing events are at risk of being\n",{"type":43,"tag":211,"props":2908,"children":2909},{"class":213,"line":1821},[2910],{"type":43,"tag":211,"props":2911,"children":2912},{},[2913],{"type":49,"value":2914},"    # lost otherwise, in short-lived scripts and long-running services alike.\n",{"type":43,"tag":211,"props":2916,"children":2917},{"class":213,"line":1830},[2918],{"type":43,"tag":211,"props":2919,"children":2920},{},[2921],{"type":49,"value":2922},"    ldclient.get().flush()\n",{"type":43,"tag":211,"props":2924,"children":2925},{"class":213,"line":1839},[2926],{"type":43,"tag":211,"props":2927,"children":2928},{},[2929],{"type":49,"value":2930},"    ldclient.get().close()\n",{"type":43,"tag":58,"props":2932,"children":2934},{"id":2933},"sdk-direct-judge-evaluation",[2935],{"type":49,"value":2936},"SDK: Direct Judge Evaluation",{"type":43,"tag":52,"props":2938,"children":2939},{},[2940],{"type":49,"value":2941},"For agent mode or custom pipelines, evaluate input\u002Foutput pairs directly:",{"type":43,"tag":200,"props":2943,"children":2945},{"className":1409,"code":2944,"language":1411,"meta":205,"style":205},"import os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AIJudgeConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\njudge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    judge_default_value = AIJudgeConfigDefault(enabled=False)\n\n    # Get judge configuration from LaunchDarkly\n    judge = aiclient.create_judge(judge_key, context, judge_default_value)\n\n    if not judge:\n        print(f\"agent judge configuration not enabled for key: {judge_key}\")\n        return\n\n    input_text = 'You are a helpful assistant. How can you help me?'\n    output_text = 'I can answer any question you have.'\n\n    # Evaluate the input\u002Foutput pair — returns a JudgeResult.\n    judge_result = await judge.evaluate(input_text, output_text)\n\n    if not judge_result.sampled:\n        print(\"Judge evaluation was skipped (sample rate or configuration issue)\")\n        return\n\n    # Track the consolidated result on the Config tracker if needed:\n    # tracker = ai_config.create_tracker()\n    # tracker.track_judge_result(judge_result)\n\n    print(\"Judge Result:\")\n    print(json.dumps(judge_result.to_dict(), default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n",[2946],{"type":43,"tag":85,"props":2947,"children":2948},{"__ignoreMap":205},[2949,2956,2963,2970,2977,2984,2991,2999,3006,3013,3021,3028,3035,3042,3049,3056,3063,3070,3077,3084,3091,3098,3105,3113,3120,3128,3136,3143,3151,3159,3166,3173,3181,3189,3196,3204,3212,3219,3227,3235,3242,3249,3257,3265,3273,3280,3288,3296,3303,3310,3317,3324],{"type":43,"tag":211,"props":2950,"children":2951},{"class":213,"line":214},[2952],{"type":43,"tag":211,"props":2953,"children":2954},{},[2955],{"type":49,"value":1431},{"type":43,"tag":211,"props":2957,"children":2958},{"class":213,"line":224},[2959],{"type":43,"tag":211,"props":2960,"children":2961},{},[2962],{"type":49,"value":2548},{"type":43,"tag":211,"props":2964,"children":2965},{"class":213,"line":260},[2966],{"type":43,"tag":211,"props":2967,"children":2968},{},[2969],{"type":49,"value":2556},{"type":43,"tag":211,"props":2971,"children":2972},{"class":213,"line":297},[2973],{"type":43,"tag":211,"props":2974,"children":2975},{},[2976],{"type":49,"value":2564},{"type":43,"tag":211,"props":2978,"children":2979},{"class":213,"line":580},[2980],{"type":43,"tag":211,"props":2981,"children":2982},{},[2983],{"type":49,"value":2572},{"type":43,"tag":211,"props":2985,"children":2986},{"class":213,"line":24},[2987],{"type":43,"tag":211,"props":2988,"children":2989},{},[2990],{"type":49,"value":2580},{"type":43,"tag":211,"props":2992,"children":2993},{"class":213,"line":622},[2994],{"type":43,"tag":211,"props":2995,"children":2996},{},[2997],{"type":49,"value":2998},"from ldai import LDAIClient, AIJudgeConfigDefault\n",{"type":43,"tag":211,"props":3000,"children":3001},{"class":213,"line":631},[3002],{"type":43,"tag":211,"props":3003,"children":3004},{"emptyLinePlaceholder":1445},[3005],{"type":49,"value":1448},{"type":43,"tag":211,"props":3007,"children":3008},{"class":213,"line":640},[3009],{"type":43,"tag":211,"props":3010,"children":3011},{},[3012],{"type":49,"value":2603},{"type":43,"tag":211,"props":3014,"children":3015},{"class":213,"line":649},[3016],{"type":43,"tag":211,"props":3017,"children":3018},{},[3019],{"type":49,"value":3020},"judge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy')\n",{"type":43,"tag":211,"props":3022,"children":3023},{"class":213,"line":658},[3024],{"type":43,"tag":211,"props":3025,"children":3026},{"emptyLinePlaceholder":1445},[3027],{"type":49,"value":1448},{"type":43,"tag":211,"props":3029,"children":3030},{"class":213,"line":667},[3031],{"type":43,"tag":211,"props":3032,"children":3033},{},[3034],{"type":49,"value":2626},{"type":43,"tag":211,"props":3036,"children":3037},{"class":213,"line":887},[3038],{"type":43,"tag":211,"props":3039,"children":3040},{},[3041],{"type":49,"value":2634},{"type":43,"tag":211,"props":3043,"children":3044},{"class":213,"line":896},[3045],{"type":43,"tag":211,"props":3046,"children":3047},{},[3048],{"type":49,"value":2642},{"type":43,"tag":211,"props":3050,"children":3051},{"class":213,"line":905},[3052],{"type":43,"tag":211,"props":3053,"children":3054},{"emptyLinePlaceholder":1445},[3055],{"type":49,"value":1448},{"type":43,"tag":211,"props":3057,"children":3058},{"class":213,"line":914},[3059],{"type":43,"tag":211,"props":3060,"children":3061},{},[3062],{"type":49,"value":2657},{"type":43,"tag":211,"props":3064,"children":3065},{"class":213,"line":923},[3066],{"type":43,"tag":211,"props":3067,"children":3068},{},[3069],{"type":49,"value":2665},{"type":43,"tag":211,"props":3071,"children":3072},{"class":213,"line":932},[3073],{"type":43,"tag":211,"props":3074,"children":3075},{},[3076],{"type":49,"value":2673},{"type":43,"tag":211,"props":3078,"children":3079},{"class":213,"line":940},[3080],{"type":43,"tag":211,"props":3081,"children":3082},{},[3083],{"type":49,"value":2681},{"type":43,"tag":211,"props":3085,"children":3086},{"class":213,"line":20},[3087],{"type":43,"tag":211,"props":3088,"children":3089},{},[3090],{"type":49,"value":2689},{"type":43,"tag":211,"props":3092,"children":3093},{"class":213,"line":1577},[3094],{"type":43,"tag":211,"props":3095,"children":3096},{},[3097],{"type":49,"value":2697},{"type":43,"tag":211,"props":3099,"children":3100},{"class":213,"line":1586},[3101],{"type":43,"tag":211,"props":3102,"children":3103},{"emptyLinePlaceholder":1445},[3104],{"type":49,"value":1448},{"type":43,"tag":211,"props":3106,"children":3107},{"class":213,"line":1594},[3108],{"type":43,"tag":211,"props":3109,"children":3110},{},[3111],{"type":49,"value":3112},"    judge_default_value = AIJudgeConfigDefault(enabled=False)\n",{"type":43,"tag":211,"props":3114,"children":3115},{"class":213,"line":1603},[3116],{"type":43,"tag":211,"props":3117,"children":3118},{"emptyLinePlaceholder":1445},[3119],{"type":49,"value":1448},{"type":43,"tag":211,"props":3121,"children":3122},{"class":213,"line":1612},[3123],{"type":43,"tag":211,"props":3124,"children":3125},{},[3126],{"type":49,"value":3127},"    # Get judge configuration from LaunchDarkly\n",{"type":43,"tag":211,"props":3129,"children":3130},{"class":213,"line":1621},[3131],{"type":43,"tag":211,"props":3132,"children":3133},{},[3134],{"type":49,"value":3135},"    judge = aiclient.create_judge(judge_key, context, judge_default_value)\n",{"type":43,"tag":211,"props":3137,"children":3138},{"class":213,"line":1630},[3139],{"type":43,"tag":211,"props":3140,"children":3141},{"emptyLinePlaceholder":1445},[3142],{"type":49,"value":1448},{"type":43,"tag":211,"props":3144,"children":3145},{"class":213,"line":1638},[3146],{"type":43,"tag":211,"props":3147,"children":3148},{},[3149],{"type":49,"value":3150},"    if not judge:\n",{"type":43,"tag":211,"props":3152,"children":3153},{"class":213,"line":1647},[3154],{"type":43,"tag":211,"props":3155,"children":3156},{},[3157],{"type":49,"value":3158},"        print(f\"agent judge configuration not enabled for key: {judge_key}\")\n",{"type":43,"tag":211,"props":3160,"children":3161},{"class":213,"line":1655},[3162],{"type":43,"tag":211,"props":3163,"children":3164},{},[3165],{"type":49,"value":2766},{"type":43,"tag":211,"props":3167,"children":3168},{"class":213,"line":1664},[3169],{"type":43,"tag":211,"props":3170,"children":3171},{"emptyLinePlaceholder":1445},[3172],{"type":49,"value":1448},{"type":43,"tag":211,"props":3174,"children":3175},{"class":213,"line":1673},[3176],{"type":43,"tag":211,"props":3177,"children":3178},{},[3179],{"type":49,"value":3180},"    input_text = 'You are a helpful assistant. How can you help me?'\n",{"type":43,"tag":211,"props":3182,"children":3183},{"class":213,"line":1682},[3184],{"type":43,"tag":211,"props":3185,"children":3186},{},[3187],{"type":49,"value":3188},"    output_text = 'I can answer any question you have.'\n",{"type":43,"tag":211,"props":3190,"children":3191},{"class":213,"line":1690},[3192],{"type":43,"tag":211,"props":3193,"children":3194},{"emptyLinePlaceholder":1445},[3195],{"type":49,"value":1448},{"type":43,"tag":211,"props":3197,"children":3198},{"class":213,"line":1699},[3199],{"type":43,"tag":211,"props":3200,"children":3201},{},[3202],{"type":49,"value":3203},"    # Evaluate the input\u002Foutput pair — returns a JudgeResult.\n",{"type":43,"tag":211,"props":3205,"children":3206},{"class":213,"line":1708},[3207],{"type":43,"tag":211,"props":3208,"children":3209},{},[3210],{"type":49,"value":3211},"    judge_result = await judge.evaluate(input_text, output_text)\n",{"type":43,"tag":211,"props":3213,"children":3214},{"class":213,"line":1717},[3215],{"type":43,"tag":211,"props":3216,"children":3217},{"emptyLinePlaceholder":1445},[3218],{"type":49,"value":1448},{"type":43,"tag":211,"props":3220,"children":3221},{"class":213,"line":1726},[3222],{"type":43,"tag":211,"props":3223,"children":3224},{},[3225],{"type":49,"value":3226},"    if not judge_result.sampled:\n",{"type":43,"tag":211,"props":3228,"children":3229},{"class":213,"line":1735},[3230],{"type":43,"tag":211,"props":3231,"children":3232},{},[3233],{"type":49,"value":3234},"        print(\"Judge evaluation was skipped (sample rate or configuration issue)\")\n",{"type":43,"tag":211,"props":3236,"children":3237},{"class":213,"line":1743},[3238],{"type":43,"tag":211,"props":3239,"children":3240},{},[3241],{"type":49,"value":2766},{"type":43,"tag":211,"props":3243,"children":3244},{"class":213,"line":1752},[3245],{"type":43,"tag":211,"props":3246,"children":3247},{"emptyLinePlaceholder":1445},[3248],{"type":49,"value":1448},{"type":43,"tag":211,"props":3250,"children":3251},{"class":213,"line":1761},[3252],{"type":43,"tag":211,"props":3253,"children":3254},{},[3255],{"type":49,"value":3256},"    # Track the consolidated result on the Config tracker if needed:\n",{"type":43,"tag":211,"props":3258,"children":3259},{"class":213,"line":1770},[3260],{"type":43,"tag":211,"props":3261,"children":3262},{},[3263],{"type":49,"value":3264},"    # tracker = ai_config.create_tracker()\n",{"type":43,"tag":211,"props":3266,"children":3267},{"class":213,"line":1778},[3268],{"type":43,"tag":211,"props":3269,"children":3270},{},[3271],{"type":49,"value":3272},"    # tracker.track_judge_result(judge_result)\n",{"type":43,"tag":211,"props":3274,"children":3275},{"class":213,"line":1787},[3276],{"type":43,"tag":211,"props":3277,"children":3278},{"emptyLinePlaceholder":1445},[3279],{"type":49,"value":1448},{"type":43,"tag":211,"props":3281,"children":3282},{"class":213,"line":1795},[3283],{"type":43,"tag":211,"props":3284,"children":3285},{},[3286],{"type":49,"value":3287},"    print(\"Judge Result:\")\n",{"type":43,"tag":211,"props":3289,"children":3290},{"class":213,"line":1803},[3291],{"type":43,"tag":211,"props":3292,"children":3293},{},[3294],{"type":49,"value":3295},"    print(json.dumps(judge_result.to_dict(), default=str))\n",{"type":43,"tag":211,"props":3297,"children":3298},{"class":213,"line":1812},[3299],{"type":43,"tag":211,"props":3300,"children":3301},{"emptyLinePlaceholder":1445},[3302],{"type":49,"value":1448},{"type":43,"tag":211,"props":3304,"children":3305},{"class":213,"line":1821},[3306],{"type":43,"tag":211,"props":3307,"children":3308},{},[3309],{"type":49,"value":2906},{"type":43,"tag":211,"props":3311,"children":3312},{"class":213,"line":1830},[3313],{"type":43,"tag":211,"props":3314,"children":3315},{},[3316],{"type":49,"value":2914},{"type":43,"tag":211,"props":3318,"children":3319},{"class":213,"line":1839},[3320],{"type":43,"tag":211,"props":3321,"children":3322},{},[3323],{"type":49,"value":2922},{"type":43,"tag":211,"props":3325,"children":3326},{"class":213,"line":1848},[3327],{"type":43,"tag":211,"props":3328,"children":3329},{},[3330],{"type":49,"value":2930},{"type":43,"tag":680,"props":3332,"children":3333},{},[3334],{"type":43,"tag":52,"props":3335,"children":3336},{},[3337,3341,3343,3349,3351,3357,3359,3365,3366,3372],{"type":43,"tag":112,"props":3338,"children":3339},{},[3340],{"type":49,"value":690},{"type":49,"value":3342}," Direct evaluation does not automatically record metrics. Obtain a tracker via ",{"type":43,"tag":85,"props":3344,"children":3346},{"className":3345},[],[3347],{"type":49,"value":3348},"ai_config.create_tracker()",{"type":49,"value":3350}," \u002F ",{"type":43,"tag":85,"props":3352,"children":3354},{"className":3353},[],[3355],{"type":49,"value":3356},"aiConfig.createTracker()",{"type":49,"value":3358}," and call ",{"type":43,"tag":85,"props":3360,"children":3362},{"className":3361},[],[3363],{"type":49,"value":3364},"tracker.track_judge_result(result)",{"type":49,"value":3350},{"type":43,"tag":85,"props":3367,"children":3369},{"className":3368},[],[3370],{"type":49,"value":3371},"tracker.trackJudgeResult(result)",{"type":49,"value":3373}," to record scores for the config you're evaluating.",{"type":43,"tag":58,"props":3375,"children":3377},{"id":3376},"sampling-rates",[3378],{"type":49,"value":3379},"Sampling Rates",{"type":43,"tag":52,"props":3381,"children":3382},{},[3383],{"type":49,"value":3384},"Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.",{"type":43,"tag":52,"props":3386,"children":3387},{},[3388],{"type":49,"value":3389},"You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.",{"type":43,"tag":58,"props":3391,"children":3393},{"id":3392},"viewing-results",[3394],{"type":49,"value":3395},"Viewing Results",{"type":43,"tag":105,"props":3397,"children":3398},{},[3399,3411,3423,3435],{"type":43,"tag":69,"props":3400,"children":3401},{},[3402,3404,3409],{"type":49,"value":3403},"Navigate to ",{"type":43,"tag":112,"props":3405,"children":3406},{},[3407],{"type":49,"value":3408},"configs",{"type":49,"value":3410}," > select your config",{"type":43,"tag":69,"props":3412,"children":3413},{},[3414,3416,3421],{"type":49,"value":3415},"Click ",{"type":43,"tag":112,"props":3417,"children":3418},{},[3419],{"type":49,"value":3420},"Monitoring",{"type":49,"value":3422}," tab",{"type":43,"tag":69,"props":3424,"children":3425},{},[3426,3428,3433],{"type":49,"value":3427},"Select ",{"type":43,"tag":112,"props":3429,"children":3430},{},[3431],{"type":49,"value":3432},"Evaluator metrics",{"type":49,"value":3434}," from dropdown",{"type":43,"tag":69,"props":3436,"children":3437},{},[3438],{"type":49,"value":3439},"View scores by variation and time range",{"type":43,"tag":52,"props":3441,"children":3442},{},[3443],{"type":49,"value":3444},"Results appear within 1-2 minutes of evaluation.",{"type":43,"tag":58,"props":3446,"children":3448},{"id":3447},"use-in-guardrails-and-experiments",[3449],{"type":49,"value":3450},"Use in Guardrails and Experiments",{"type":43,"tag":52,"props":3452,"children":3453},{},[3454],{"type":49,"value":3455},"Evaluation metrics integrate with:",{"type":43,"tag":65,"props":3457,"children":3458},{},[3459,3469],{"type":43,"tag":69,"props":3460,"children":3461},{},[3462,3467],{"type":43,"tag":112,"props":3463,"children":3464},{},[3465],{"type":49,"value":3466},"Guarded rollouts",{"type":49,"value":3468},": Pause\u002Frevert when scores fall below threshold",{"type":43,"tag":69,"props":3470,"children":3471},{},[3472,3477],{"type":43,"tag":112,"props":3473,"children":3474},{},[3475],{"type":49,"value":3476},"Experiments",{"type":49,"value":3478},": Compare variations using evaluation metrics as goals",{"type":43,"tag":58,"props":3480,"children":3482},{"id":3481},"error-handling",[3483],{"type":49,"value":3484},"Error Handling",{"type":43,"tag":316,"props":3486,"children":3487},{},[3488,3509],{"type":43,"tag":320,"props":3489,"children":3490},{},[3491],{"type":43,"tag":324,"props":3492,"children":3493},{},[3494,3499,3504],{"type":43,"tag":328,"props":3495,"children":3496},{},[3497],{"type":49,"value":3498},"Status",{"type":43,"tag":328,"props":3500,"children":3501},{},[3502],{"type":49,"value":3503},"Cause",{"type":43,"tag":328,"props":3505,"children":3506},{},[3507],{"type":49,"value":3508},"Solution",{"type":43,"tag":344,"props":3510,"children":3511},{},[3512,3530,3548,3566],{"type":43,"tag":324,"props":3513,"children":3514},{},[3515,3520,3525],{"type":43,"tag":351,"props":3516,"children":3517},{},[3518],{"type":49,"value":3519},"404",{"type":43,"tag":351,"props":3521,"children":3522},{},[3523],{"type":49,"value":3524},"Config\u002Fvariation not found",{"type":43,"tag":351,"props":3526,"children":3527},{},[3528],{"type":49,"value":3529},"Verify keys exist",{"type":43,"tag":324,"props":3531,"children":3532},{},[3533,3538,3543],{"type":43,"tag":351,"props":3534,"children":3535},{},[3536],{"type":49,"value":3537},"400",{"type":43,"tag":351,"props":3539,"children":3540},{},[3541],{"type":49,"value":3542},"Invalid judge config",{"type":43,"tag":351,"props":3544,"children":3545},{},[3546],{"type":49,"value":3547},"Check judgeConfigKey exists",{"type":43,"tag":324,"props":3549,"children":3550},{},[3551,3556,3561],{"type":43,"tag":351,"props":3552,"children":3553},{},[3554],{"type":49,"value":3555},"403",{"type":43,"tag":351,"props":3557,"children":3558},{},[3559],{"type":49,"value":3560},"Insufficient permissions",{"type":43,"tag":351,"props":3562,"children":3563},{},[3564],{"type":49,"value":3565},"Check API token permissions",{"type":43,"tag":324,"props":3567,"children":3568},{},[3569,3574,3579],{"type":43,"tag":351,"props":3570,"children":3571},{},[3572],{"type":49,"value":3573},"422",{"type":43,"tag":351,"props":3575,"children":3576},{},[3577],{"type":49,"value":3578},"Duplicate metric key",{"type":43,"tag":351,"props":3580,"children":3581},{},[3582],{"type":49,"value":3583},"Cannot attach multiple judges with same metric key",{"type":43,"tag":58,"props":3585,"children":3587},{"id":3586},"next-steps",[3588],{"type":49,"value":3589},"Next Steps",{"type":43,"tag":52,"props":3591,"children":3592},{},[3593],{"type":49,"value":3594},"After attaching judges:",{"type":43,"tag":105,"props":3596,"children":3597},{},[3598,3608,3618,3628],{"type":43,"tag":69,"props":3599,"children":3600},{},[3601,3606],{"type":43,"tag":112,"props":3602,"children":3603},{},[3604],{"type":49,"value":3605},"Set fallthrough",{"type":49,"value":3607}," on judge configs to an enabled variation (required)",{"type":43,"tag":69,"props":3609,"children":3610},{},[3611,3616],{"type":43,"tag":112,"props":3612,"children":3613},{},[3614],{"type":49,"value":3615},"Monitor results",{"type":49,"value":3617}," in Monitoring tab",{"type":43,"tag":69,"props":3619,"children":3620},{},[3621,3626],{"type":43,"tag":112,"props":3622,"children":3623},{},[3624],{"type":49,"value":3625},"Adjust sampling",{"type":49,"value":3627}," based on cost\u002Fcoverage needs",{"type":43,"tag":69,"props":3629,"children":3630},{},[3631,3636],{"type":43,"tag":112,"props":3632,"children":3633},{},[3634],{"type":49,"value":3635},"Set up guarded rollouts",{"type":49,"value":3637}," for automatic regression detection",{"type":43,"tag":58,"props":3639,"children":3641},{"id":3640},"related-skills",[3642],{"type":49,"value":3643},"Related Skills",{"type":43,"tag":65,"props":3645,"children":3646},{},[3647,3657,3668],{"type":43,"tag":69,"props":3648,"children":3649},{},[3650,3655],{"type":43,"tag":85,"props":3651,"children":3653},{"className":3652},[],[3654],{"type":49,"value":90},{"type":49,"value":3656}," - Create configs and judges",{"type":43,"tag":69,"props":3658,"children":3659},{},[3660,3666],{"type":43,"tag":85,"props":3661,"children":3663},{"className":3662},[],[3664],{"type":49,"value":3665},"configs-targeting",{"type":49,"value":3667}," - Configure targeting rules",{"type":43,"tag":69,"props":3669,"children":3670},{},[3671,3677],{"type":43,"tag":85,"props":3672,"children":3674},{"className":3673},[],[3675],{"type":49,"value":3676},"configs-variations",{"type":49,"value":3678}," - Manage variations",{"type":43,"tag":58,"props":3680,"children":3682},{"id":3681},"references",[3683],{"type":49,"value":3684},"References",{"type":43,"tag":65,"props":3686,"children":3687},{},[3688,3700],{"type":43,"tag":69,"props":3689,"children":3690},{},[3691],{"type":43,"tag":3692,"props":3693,"children":3697},"a",{"href":3694,"rel":3695},"https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fonline-evaluations.md",[3696],"nofollow",[3698],{"type":49,"value":3699},"Online Evaluations",{"type":43,"tag":69,"props":3701,"children":3702},{},[3703],{"type":43,"tag":3692,"props":3704,"children":3707},{"href":3705,"rel":3706},"https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fcustom-judges.md",[3696],[3708],{"type":49,"value":3709},"Custom Judges",{"type":43,"tag":52,"props":3711,"children":3712},{},[3713],{"type":43,"tag":112,"props":3714,"children":3715},{},[3716],{"type":49,"value":3717},"Python SDK examples:",{"type":43,"tag":65,"props":3719,"children":3720},{},[3721,3746],{"type":43,"tag":69,"props":3722,"children":3723},{},[3724,3731,3733,3739,3740],{"type":43,"tag":3692,"props":3725,"children":3728},{"href":3726,"rel":3727},"https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_judge\u002Fcreate_judge_example.py",[3696],[3729],{"type":49,"value":3730},"create_judge_example.py",{"type":49,"value":3732}," - Evaluate input\u002Foutput pairs directly via ",{"type":43,"tag":85,"props":3734,"children":3736},{"className":3735},[],[3737],{"type":49,"value":3738},"create_judge",{"type":49,"value":2518},{"type":43,"tag":85,"props":3741,"children":3743},{"className":3742},[],[3744],{"type":49,"value":3745},"evaluate",{"type":43,"tag":69,"props":3747,"children":3748},{},[3749,3756,3758,3764,3765,3771],{"type":43,"tag":3692,"props":3750,"children":3753},{"href":3751,"rel":3752},"https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_model\u002Fcreate_model_example.py",[3696],[3754],{"type":49,"value":3755},"create_model_example.py",{"type":49,"value":3757}," - Automatic evaluation with ",{"type":43,"tag":85,"props":3759,"children":3761},{"className":3760},[],[3762],{"type":49,"value":3763},"create_model",{"type":49,"value":2518},{"type":43,"tag":85,"props":3766,"children":3768},{"className":3767},[],[3769],{"type":49,"value":3770},"run",{"type":49,"value":3772}," (attached judges fire during the run)",{"type":43,"tag":52,"props":3774,"children":3775},{},[3776],{"type":43,"tag":112,"props":3777,"children":3778},{},[3779],{"type":49,"value":3780},"Node.js SDK examples:",{"type":43,"tag":65,"props":3782,"children":3783},{},[3784,3807],{"type":43,"tag":69,"props":3785,"children":3786},{},[3787,3794,3795,3801,3802],{"type":43,"tag":3692,"props":3788,"children":3791},{"href":3789,"rel":3790},"https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-judge\u002Fsrc\u002Findex.ts",[3696],[3792],{"type":49,"value":3793},"features\u002Fcreate-judge",{"type":49,"value":3732},{"type":43,"tag":85,"props":3796,"children":3798},{"className":3797},[],[3799],{"type":49,"value":3800},"createJudge",{"type":49,"value":2518},{"type":43,"tag":85,"props":3803,"children":3805},{"className":3804},[],[3806],{"type":49,"value":3745},{"type":43,"tag":69,"props":3808,"children":3809},{},[3810,3817,3818,3824,3825,3830],{"type":43,"tag":3692,"props":3811,"children":3814},{"href":3812,"rel":3813},"https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-model\u002Fsrc\u002Findex.ts",[3696],[3815],{"type":49,"value":3816},"features\u002Fcreate-model",{"type":49,"value":3757},{"type":43,"tag":85,"props":3819,"children":3821},{"className":3820},[],[3822],{"type":49,"value":3823},"createModel",{"type":49,"value":2518},{"type":43,"tag":85,"props":3826,"children":3828},{"className":3827},[],[3829],{"type":49,"value":3770},{"type":49,"value":3772},{"type":43,"tag":3832,"props":3833,"children":3834},"style",{},[3835],{"type":49,"value":3836},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3838,"total":1821},[3839,3856,3865,3879,3890,3902,3910,3922,3933,3942,3952,3961],{"slug":3840,"name":3840,"fn":3841,"description":3842,"org":3843,"tags":3844,"stars":20,"repoUrl":21,"updatedAt":3855},"agent-graphs","create and manage agent graphs","Create and manage agent graphs — directed graphs of configs connected by edges with handoff logic. Use when building multi-agent workflows where configs route to each other.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3845,3848,3851,3852],{"name":3846,"slug":3847,"type":15},"Agents","agents",{"name":3849,"slug":3850,"type":15},"Architecture","architecture",{"name":9,"slug":8,"type":15},{"name":3853,"slug":3854,"type":15},"Multi-Agent","multi-agent","2026-07-28T05:33:33.709407",{"slug":3857,"name":3857,"fn":3858,"description":3859,"org":3860,"tags":3861,"stars":20,"repoUrl":21,"updatedAt":3864},"aiconfig-agent-graphs","manage agent graphs","DEPRECATED redirect — this skill was renamed to agent-graphs. Do not use this skill; invoke agent-graphs instead. Kept only so old references to aiconfig-agent-graphs still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3862,3863],{"name":3846,"slug":3847,"type":15},{"name":3849,"slug":3850,"type":15},"2026-05-22T06:55:56.527064",{"slug":3866,"name":3866,"fn":3867,"description":3868,"org":3869,"tags":3870,"stars":20,"repoUrl":21,"updatedAt":3878},"aiconfig-ai-metrics","manage built-in AI metrics","DEPRECATED redirect — this skill was renamed to built-in-metrics. Do not use this skill; invoke built-in-metrics instead. Kept only so old references to aiconfig-ai-metrics still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3871,3874,3875],{"name":3872,"slug":3873,"type":15},"Analytics","analytics",{"name":9,"slug":8,"type":15},{"name":3876,"slug":3877,"type":15},"Metrics","metrics","2026-05-22T06:55:53.858749",{"slug":3880,"name":3880,"fn":3881,"description":3882,"org":3883,"tags":3884,"stars":20,"repoUrl":21,"updatedAt":3889},"aiconfig-create","redirect to configs-create skill","DEPRECATED redirect — this skill was renamed to configs-create. Do not use this skill; invoke configs-create instead. Kept only so old references to aiconfig-create still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3885,3886],{"name":9,"slug":8,"type":15},{"name":3887,"slug":3888,"type":15},"Reference","reference","2026-05-22T06:55:41.790591",{"slug":3891,"name":3891,"fn":3892,"description":3893,"org":3894,"tags":3895,"stars":20,"repoUrl":21,"updatedAt":3901},"aiconfig-custom-metrics","configure custom metrics in LaunchDarkly","DEPRECATED redirect — this skill was renamed to custom-metrics. Do not use this skill; invoke custom-metrics instead. Kept only so old references to aiconfig-custom-metrics still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3896,3897,3900],{"name":3872,"slug":3873,"type":15},{"name":3898,"slug":3899,"type":15},"Feature Flags","feature-flags",{"name":9,"slug":8,"type":15},"2026-05-22T06:55:57.84851",{"slug":3903,"name":3903,"fn":3904,"description":3905,"org":3906,"tags":3907,"stars":20,"repoUrl":21,"updatedAt":3909},"aiconfig-migrate","redirect to migrate skill","DEPRECATED redirect — this skill was renamed to migrate. Do not use this skill; invoke migrate instead. Kept only so old references to aiconfig-migrate still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3908],{"name":3887,"slug":3888,"type":15},"2026-05-22T06:55:44.464733",{"slug":3911,"name":3911,"fn":3912,"description":3913,"org":3914,"tags":3915,"stars":20,"repoUrl":21,"updatedAt":3921},"aiconfig-online-evals","run online evaluations","DEPRECATED redirect — this skill was renamed to online-evals. Do not use this skill; invoke online-evals instead. Kept only so old references to aiconfig-online-evals still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3916,3917,3918],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3919,"slug":3920,"type":15},"Testing","testing","2026-05-22T06:55:55.179617",{"slug":3923,"name":3923,"fn":3924,"description":3925,"org":3926,"tags":3927,"stars":20,"repoUrl":21,"updatedAt":3932},"aiconfig-projects","manage AI configuration projects","DEPRECATED redirect — this skill was renamed to projects. Do not use this skill; invoke projects instead. Kept only so old references to aiconfig-projects still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3928,3931],{"name":3929,"slug":3930,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},"2026-05-22T06:55:48.522229",{"slug":3934,"name":3934,"fn":3935,"description":3936,"org":3937,"tags":3938,"stars":20,"repoUrl":21,"updatedAt":3941},"aiconfig-snippets","manage AI configuration snippets","DEPRECATED redirect — this skill was renamed to snippets. Do not use this skill; invoke snippets instead. Kept only so old references to aiconfig-snippets still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3939,3940],{"name":3929,"slug":3930,"type":15},{"name":9,"slug":8,"type":15},"2026-05-22T06:55:47.16557",{"slug":3943,"name":3943,"fn":3944,"description":3945,"org":3946,"tags":3947,"stars":20,"repoUrl":21,"updatedAt":3951},"aiconfig-targeting","configure LaunchDarkly targeting rules","DEPRECATED redirect — this skill was renamed to configs-targeting. Do not use this skill; invoke configs-targeting instead. Kept only so old references to aiconfig-targeting still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3948,3949,3950],{"name":3929,"slug":3930,"type":15},{"name":3898,"slug":3899,"type":15},{"name":9,"slug":8,"type":15},"2026-05-22T06:55:49.845445",{"slug":3953,"name":3953,"fn":3954,"description":3955,"org":3956,"tags":3957,"stars":20,"repoUrl":21,"updatedAt":3960},"aiconfig-tools","redirect to tools skill","DEPRECATED redirect — this skill was renamed to tools. Do not use this skill; invoke tools instead. Kept only so old references to aiconfig-tools still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3958,3959],{"name":9,"slug":8,"type":15},{"name":3887,"slug":3888,"type":15},"2026-05-22T06:55:39.13373",{"slug":3962,"name":3962,"fn":3963,"description":3964,"org":3965,"tags":3966,"stars":20,"repoUrl":21,"updatedAt":3969},"aiconfig-update","redirect to configs-update skill","DEPRECATED redirect — this skill was renamed to configs-update. Do not use this skill; invoke configs-update instead. Kept only so old references to aiconfig-update still point users to the new name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3967,3968],{"name":9,"slug":8,"type":15},{"name":3887,"slug":3888,"type":15},"2026-05-22T06:55:40.464884",{"items":3971,"total":1821},[3972,3979,3984,3990,3995,4001,4005],{"slug":3840,"name":3840,"fn":3841,"description":3842,"org":3973,"tags":3974,"stars":20,"repoUrl":21,"updatedAt":3855},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3975,3976,3977,3978],{"name":3846,"slug":3847,"type":15},{"name":3849,"slug":3850,"type":15},{"name":9,"slug":8,"type":15},{"name":3853,"slug":3854,"type":15},{"slug":3857,"name":3857,"fn":3858,"description":3859,"org":3980,"tags":3981,"stars":20,"repoUrl":21,"updatedAt":3864},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3982,3983],{"name":3846,"slug":3847,"type":15},{"name":3849,"slug":3850,"type":15},{"slug":3866,"name":3866,"fn":3867,"description":3868,"org":3985,"tags":3986,"stars":20,"repoUrl":21,"updatedAt":3878},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3987,3988,3989],{"name":3872,"slug":3873,"type":15},{"name":9,"slug":8,"type":15},{"name":3876,"slug":3877,"type":15},{"slug":3880,"name":3880,"fn":3881,"description":3882,"org":3991,"tags":3992,"stars":20,"repoUrl":21,"updatedAt":3889},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3993,3994],{"name":9,"slug":8,"type":15},{"name":3887,"slug":3888,"type":15},{"slug":3891,"name":3891,"fn":3892,"description":3893,"org":3996,"tags":3997,"stars":20,"repoUrl":21,"updatedAt":3901},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3998,3999,4000],{"name":3872,"slug":3873,"type":15},{"name":3898,"slug":3899,"type":15},{"name":9,"slug":8,"type":15},{"slug":3903,"name":3903,"fn":3904,"description":3905,"org":4002,"tags":4003,"stars":20,"repoUrl":21,"updatedAt":3909},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4004],{"name":3887,"slug":3888,"type":15},{"slug":3911,"name":3911,"fn":3912,"description":3913,"org":4006,"tags":4007,"stars":20,"repoUrl":21,"updatedAt":3921},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4008,4009,4010],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3919,"slug":3920,"type":15}]