[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-azure-agentops-eval":3,"mdc--93ovgd-key":32,"related-org-azure-agentops-eval":3079,"related-repo-azure-agentops-eval":3260},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":21,"repoUrl":22,"updatedAt":23,"license":24,"forks":25,"topics":26,"repo":27,"sourceUrl":30,"mdContent":31},"agentops-eval","run release-readiness evaluations for agents","Run AgentOps release-readiness evaluations against Foundry prompt agents, Foundry hosted endpoints, HTTP\u002FJSON agents, or raw model deployments. Trigger on phrases like \"run eval\", \"evaluate my agent\", \"benchmark\", \"agentops eval\", \"compare runs\", \"can we ship\". Uses the flat agentops.yaml schema.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"azure","Azure (Microsoft)","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fazure.png","Azure",[13,15,18],{"name":11,"slug":8,"type":14},"tag",{"name":16,"slug":17,"type":14},"Evals","evals",{"name":19,"slug":20,"type":14},"Agents","agents",10,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fagentops","2026-07-15T06:02:50.097169",null,7,[],{"repoUrl":22,"stars":21,"forks":25,"topics":28,"description":29},[],"AgentOps Accelerator is an open source framework and CLI for adding continuous evaluation and observability to enterprise AI agents. It standardizes evaluation patterns, automates assessments in CI\u002FCD workflows, and generates structured signals that help teams monitor, control, and safely operate agentic systems at scale.","https:\u002F\u002Fgithub.com\u002FAzure\u002Fagentops\u002Ftree\u002FHEAD\u002Fplugins\u002Fagentops\u002Fskills\u002Fagentops-eval","---\nname: agentops-eval\ndescription: Run AgentOps release-readiness evaluations against Foundry prompt agents, Foundry hosted endpoints, HTTP\u002FJSON agents, or raw model deployments. Trigger on phrases like \"run eval\", \"evaluate my agent\", \"benchmark\", \"agentops eval\", \"compare runs\", \"can we ship\". Uses the flat agentops.yaml schema.\n---\n\n# AgentOps Eval\n\nEnd-to-end release-gate workflow: install -> init -> configure -> run -> read\nreport -> decide whether the candidate is ready to ship.\n\nAgentOps evaluates an existing candidate. It does **not** create or deploy\nFoundry agents. If the user still needs a Prompt Agent or Hosted Agent, hand off\nto Foundry Toolkit \u002F the `microsoft-foundry` skill \u002F azd first, then come back\nwith a `name:version` or URL.\n\n## Step 0 - Setup\n\n1. Install if missing: `pip install \"agentops-accelerator @ git+https:\u002F\u002Fgithub.com\u002FAzure\u002Fagentops.git@main\"`.\n2. If `agentops.yaml` does not exist at the project root, run `agentops init`.\n   The init wizard prompts (azd-style) for the Foundry project endpoint,\n   agent reference, and dataset path, persists each answer to\n   `.agentops\u002F.env` + `agentops.yaml` as it goes. Existing azd workspaces, or\n   runs with `--azd-env`, use `.azure\u002F\u003Cenv>\u002F.env` instead. Pass `--no-prompt`\n   plus the explicit flags\n   (`--project-endpoint`, `--agent`, `--dataset`, …) for non-interactive\n   runs. Run `agentops init show` later to inspect the resolved config.\n\n## Step 0.5 - Ensure agent-build and data-plane RBAC on the AI Services account\n\nAgentOps eval (cloud graders **and** local AI-assisted evaluators) calls\n`\u002Fopenai\u002Fdeployments\u002F...\u002Fchat\u002Fcompletions` on the AI Services account\nthat backs the Foundry project. Creating a project through the Foundry\nportal only assigns the user `Foundry User` at the *project* scope,\nwhich does **not** cover the parent account. The Foundry UI may then block\nagent creation with \"You don't have permission to build agents in this\nproject\" and ask for the old portal label, `Azure AI User`; the current Azure\nRBAC role is `Foundry User` (`53ca6127-db72-4b80-b1b0-d745d6d5456d`).\n\nYou also need `Cognitive Services OpenAI User`\n(`5e0bd9bd-7b93-4f28-af87-19fc36ad61bd`) for OpenAI data-plane actions on\nthe parent account. Subscription `Owner` is insufficient because the built-in\n`Owner` role has `actions: [\"*\"]` but `dataActions: []`. The first\n`agentops eval run` against a fresh workspace otherwise fails with:\n\n```\nPermissionDenied … lacks the required data action\n'Microsoft.CognitiveServices\u002Faccounts\u002FOpenAI\u002Fdeployments\u002Fchat\u002Fcompletions\u002Faction'\n```\n\nRun this preflight before Step 1. It must grant `Foundry User` and\n`Cognitive Services OpenAI User` to the signed-in user on the AI Services\naccount, plus the OpenAI data-plane role to any Foundry\u002FAzure AI managed\nidentities in the resource group. Cloud evaluations run server-side and some\ngraders authenticate as those managed identities, so assigning only the user\ncan still produce intermittent `AuthenticationError` grader failures. The\ncommands are idempotent (`RoleAssignmentExists` means the role was already\ngranted):\n\n```bash\n# 1. Resolve the AI Services account from agentops.yaml \u002F .azure\u002F\u003Cenv>\u002F.env\nPROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure\u002F*\u002F.env .agentops\u002F.env 2>\u002Fdev\u002Fnull | tail -1 | cut -d= -f2- | tr -d '\"')\nACCOUNT_HOST=$(echo \"$PROJECT_ENDPOINT\" | awk -F[\u002F:] '{print $4}')\nACCOUNT_NAME=$(echo \"$ACCOUNT_HOST\" | cut -d. -f1)\n\n# 2. Resolve subscription, resource group, account scope, and signed-in object ID\nSUB_ID=$(az account show --query id -o tsv)\nRG=$(az cognitiveservices account list --subscription \"$SUB_ID\" --query \"[?name=='$ACCOUNT_NAME'].resourceGroup | [0]\" -o tsv)\nACCOUNT_ID=$(az cognitiveservices account show -g \"$RG\" -n \"$ACCOUNT_NAME\" --query id -o tsv)\nOBJ_ID=$(az ad signed-in-user show --query id -o tsv)\n\n# 3. Grant the user portal build access and data-plane access at account scope.\naz role assignment create \\\n  --assignee \"$OBJ_ID\" \\\n  --role \"53ca6127-db72-4b80-b1b0-d745d6d5456d\" \\\n  --scope \"$ACCOUNT_ID\"\n\naz role assignment create \\\n  --assignee \"$OBJ_ID\" \\\n  --role \"5e0bd9bd-7b93-4f28-af87-19fc36ad61bd\" \\\n  --scope \"$ACCOUNT_ID\"\n\n# 4. Grant the same data-plane role to Foundry\u002FAzure AI managed identities.\naz resource list -g \"$RG\" \\\n  --query \"[?identity.principalId!=null].identity.principalId\" -o tsv |\nwhile read -r PRINCIPAL_ID; do\n  [ -z \"$PRINCIPAL_ID\" ] && continue\n  az role assignment create \\\n    --assignee-object-id \"$PRINCIPAL_ID\" \\\n    --assignee-principal-type ServicePrincipal \\\n    --role \"5e0bd9bd-7b93-4f28-af87-19fc36ad61bd\" \\\n    --scope \"$ACCOUNT_ID\"\ndone\n```\n\nPowerShell equivalent: replace `$(...)` with the PowerShell variable\nassignments shown in `docs\u002Ftutorial-prompt-agent.md`.\n\nIf the user has not run `az login` yet, do that first. If\n`az cognitiveservices account list` returns an empty RG, the AI Services\naccount lives in a different subscription - ask the user which one.\n\nSkip this step only if the user explicitly says both roles are already\nassigned on the parent AI Services account, or if the user can already build\nagents in the Foundry UI and a previous `agentops eval run` succeeded against\nthe same Foundry account.\n\n**Propagation:** data-plane role assignments do not take effect\ninstantly — allow several minutes (occasionally up to ~15) before the\nfirst eval. The cloud\u002Flocal graders authenticate per call, so if the\nuser runs an eval immediately after this preflight and sees intermittent\n`AuthenticationError` on a subset of graders plus\n`Threshold status: FAILED` while the visible thresholds are green, that\nis propagation lag (a grader **execution** failure), not a quality\nregression. Tell the user to wait a few minutes and re-run\n`agentops eval run`; do not treat it as a failing gate or start changing\nthresholds.\n\n## Step 1 - Analyze evaluation setup\n\nRun the deterministic local triage first:\n\n```bash\nagentops eval analyze\n```\n\nUse its output to decide whether the repo is ready for `agentops eval run` or\nneeds skill-assisted setup. If it recommends `agentops-config`, fix the target\nand protocol. If it recommends `agentops-dataset`, create\u002Fmap realistic JSONL\nrows. If it recommends `agentops-eval`, inspect the app scenario and evaluator\nexpectations before running.\n\n## Step 2 - Identify the agent target\n\nRead the codebase (README, entry point, env vars) and pick the right value\nfor the `agent:` field of `agentops.yaml`:\n\n| Pattern in code \u002F env | `agent:` value |\n|---|---|\n| Foundry Prompt Agent ID like `name:1` | `\"\u003Cname>:\u003Cversion>\"` |\n| Foundry Hosted Agent endpoint URL ending in `\u002Fagents\u002F...` | `\"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\u002Fagents\u002F...\"` |\n| Plain HTTP\u002FJSON endpoint (FastAPI, Express, ACA, AKS) | `\"https:\u002F\u002F\u003Chost>\u002F\u003Cpath>\"` |\n| Raw Foundry\u002FAzure OpenAI model deployment | `\"model:\u003Cdeployment-name>\"` |\n\nIf nothing is found, ask the user once for the agent identifier.\n\nFor Foundry Prompt Agents authored in the Sandbox portal, do not copy\u002Fpaste the\ninstructions into a file manually. After `agentops.yaml` contains `agent:\nname:version` and the correct project endpoint is available from\n`agentops.yaml`, `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`, or the active\n`.azure\u002F\u003Cenv>\u002F.env`, run:\n\n```bash\nagentops prompt pull\n```\n\nThis writes `.agentops\u002Fprompts\u002F\u003Cagent-name>.prompt.md` by default, updates\n`prompt_file` in `agentops.yaml` when needed, prints the resolved endpoint and\nagent version before writing, validates that the Foundry definition is a prompt\nagent, and refuses to overwrite changed prompt files unless `--force` is used.\nUse `--out` only when the repository already has a stronger prompt-file\nconvention.\n\n## Step 3 - Make sure the dataset exists\n\n`agentops.yaml` points to a JSONL file (default\n`.agentops\u002Fdata\u002Fsmoke.jsonl`). Each row needs at least `input` and a label\nthat maps to the metric you care about (`expected`, `context`,\n`tool_calls`...). If the dataset is empty or unrelated, run the\n`agentops-dataset` skill before running the eval.\n\n## Step 4 - Run the evaluation\n\n```bash\nagentops eval run\n```\n\nOptional flags:\n\n- `--config \u003Cpath>` - point at a different `agentops.yaml`.\n- `--output \u003Cdir>` - choose where to write `results.json` and `report.md`\n  (defaults to `.agentops\u002Fresults\u002F\u003Ctimestamp>\u002F`).\n\nExit codes:\n\n- `0` - succeeded and all thresholds passed\n- `2` - succeeded but at least one threshold failed (gate-friendly)\n- `1` - runtime\u002Fconfiguration error\n\n## Step 4b - Pick the right execution path\n\n| Target | Foundry server-side eval through AgentOps | AgentOps local runner | Default guidance |\n|---|---|---|---|\n| Foundry Prompt Agent (`name:version`) | `execution: cloud` | yes | Use cloud when the user wants the official Foundry-hosted eval record; use local for fast feedback or fallback. |\n| Foundry Hosted Agent URL | no | yes | Use local runner; optionally set `publish: true` to upload local metrics to Classic Foundry. |\n| Generic HTTP\u002FJSON endpoint | no | yes | Use local runner. |\n| Raw model deployment | no | yes | Use local runner. |\n| azd `eval.yaml` recipe | `execution: azd` | no fallback | Use `agentops eval init` \u002F `azd ai agent eval` for Foundry prompt\u002Fhosted agents when the user wants the public-preview azd AI agent eval workflow. |\n\nFor prompt-agent CI gates, prefer AgentOps cloud eval because Foundry executes\nthe managed eval while AgentOps enforces thresholds and writes normalized\n`results.json` \u002F `report.md` artifacts. The official AI Agent Evaluation GitHub\nAction or Azure DevOps extension is still useful for standalone platform-native\nvalidation, but do not substitute it for the AgentOps PR gate when the user needs\nthreshold enforcement, baselines, Doctor readiness, release evidence, or local\nfallback.\n\nWhen an `eval.yaml` includes Rubric evaluator dimensions, keep thresholds in\n`agentops.yaml` aligned to the dimension metric names (for example\n`booking_accuracy: \">=0.8\"`). AgentOps binds Rubric\u002Fcustom dimensions literally\nand fails closed when a configured threshold has no matching emitted metric.\n\n## Step 5 - Inspect results and release evidence\n\n```bash\nagentops report generate                   # regenerate report.md from latest results.json\nagentops report generate --in \u003Cresults.json>\n```\n\nOpen `.agentops\u002Fresults\u002Flatest\u002Freport.md`. To compare two runs, hand both\n`results.json` files to the user or run the next eval with\n`--baseline \u003Cprevious-results.json>` so AgentOps adds a **Comparison vs\nBaseline** section to the report.\n\nFor production promotion, generate the Doctor evidence pack:\n\n```bash\nagentops doctor --evidence-pack\n```\n\nOpen `.agentops\u002Frelease\u002Flatest\u002Fevidence.md`. It summarizes eval, baseline,\nDoctor, workflow, Foundry, monitoring, AI Landing Zone, and trace-regression\nreadiness without creating a second exit-code contract. If the repo has ASSERT,\nACS, or red-team evidence artifacts, use the `agentops-governance` skill to wire\ntheir paths into `agentops.yaml` before generating the evidence pack.\n\n## Step 5b - (Optional) Promote reviewed traces to regression rows\n\nIf the user has exported Foundry\u002FApp Insights traces, preview candidate\nregression rows first:\n\n```bash\nagentops eval promote-traces --source \u003Ctraces.jsonl>\n```\n\nOnly write files after review:\n\n```bash\nagentops eval promote-traces --source \u003Ctraces.jsonl> --apply\n```\n\nDefault `self-similarity` labels are for drift detection, not human-verified\nground truth. Use `--label-mode pending` when reviewers must fill expected\nanswers before the dataset gates releases.\n\n## Step 6 - (Optional) Foundry execution \u002F visibility\n\nTwo modes are supported. Both write a deep-link into\n`.agentops\u002Fresults\u002Flatest\u002Fcloud_evaluation.json` and require\n`AZURE_AI_FOUNDRY_PROJECT_ENDPOINT` (or the inline `project_endpoint`).\n\n**Classic Foundry Evaluations panel** (works for any target kind):\nAgentOps runs locally first, then uploads the metrics it computed.\n\n```yaml\nexecution: local\npublish: true\n# project_endpoint: \"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\"\n```\n\n**New Foundry Evaluations panel** (preview): Foundry runs the agent +\nevaluators server-side via the OpenAI Evals API. Only works for\n`name:version` Foundry agents. `publish` is implicit - a cloud run is\nalways recorded by Foundry. The local JSONL remains the dataset source of\ntruth; AgentOps syncs it to Foundry Data\u002FDatasets by default and uses that\ndataset version in the Evals run.\n\n```yaml\nexecution: cloud\n# project_endpoint: \"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\"\ndataset_sync:\n  mode: auto            # sync local JSONL to Foundry Data\u002FDatasets\n```\n\nWith `execution: local` and no `publish: true`, AgentOps runs locally\nand only writes local artifacts.\n\nAfter a cloud run, inspect `.agentops\u002Fresults\u002Flatest\u002Fcloud_evaluation.json`.\nIts `dataset` block explains whether the run used inline rows or a Foundry\ndataset reference.\n\n## Tips\n\n- Evaluators are auto-selected from the agent type and dataset columns.\n  Override only when needed via the `evaluators:` block - most users do\n  not need it.\n- Set thresholds in `thresholds:` to gate CI:\n  ```yaml\n  thresholds:\n    coherence: \">=3\"\n    avg_latency_seconds: \"\u003C=10\"\n  ```\n- For HTTP\u002FJSON agents that need auth, set\n  `auth_header_env: MY_TOKEN_VAR` and AgentOps adds\n  `Authorization: Bearer $MY_TOKEN_VAR`. For a shared-secret gate, override the\n  header with `auth_header_name: X-API-KEY` and `auth_value_template: \"{token}\"`.\n- For streaming HTTP agents (e.g. an SSE `text\u002Fevent-stream` endpoint), set\n  `response_mode: sse` (each `data:` line) or `response_mode: text` (raw\n  streamed text). Use the optional `stream:` block to tune aggregation:\n  `text_field` (dot-path to the token text when `data:` lines are JSON),\n  `done_marker` (e.g. `[DONE]`), and `strip_leading_token: true` (drop a leading\n  `conversation_id` prefix). `response_mode: json` (default) is unchanged.\n",{"data":33,"body":34},{"name":4,"description":6},{"type":35,"children":36},"root",[37,45,51,81,88,197,203,268,328,340,375,1466,1486,1507,1519,1558,1564,1569,1594,1629,1635,1655,1768,1773,1814,1838,1882,1888,1944,1950,1973,1978,2033,2038,2074,2080,2268,2286,2313,2319,2388,2423,2428,2452,2479,2485,2490,2536,2541,2589,2610,2616,2643,2653,2705,2729,2794,2814,2834,2840,3073],{"type":38,"tag":39,"props":40,"children":41},"element","h1",{"id":4},[42],{"type":43,"value":44},"text","AgentOps Eval",{"type":38,"tag":46,"props":47,"children":48},"p",{},[49],{"type":43,"value":50},"End-to-end release-gate workflow: install -> init -> configure -> run -> read\nreport -> decide whether the candidate is ready to ship.",{"type":38,"tag":46,"props":52,"children":53},{},[54,56,62,64,71,73,79],{"type":43,"value":55},"AgentOps evaluates an existing candidate. It does ",{"type":38,"tag":57,"props":58,"children":59},"strong",{},[60],{"type":43,"value":61},"not",{"type":43,"value":63}," create or deploy\nFoundry agents. If the user still needs a Prompt Agent or Hosted Agent, hand off\nto Foundry Toolkit \u002F the ",{"type":38,"tag":65,"props":66,"children":68},"code",{"className":67},[],[69],{"type":43,"value":70},"microsoft-foundry",{"type":43,"value":72}," skill \u002F azd first, then come back\nwith a ",{"type":38,"tag":65,"props":74,"children":76},{"className":75},[],[77],{"type":43,"value":78},"name:version",{"type":43,"value":80}," or URL.",{"type":38,"tag":82,"props":83,"children":85},"h2",{"id":84},"step-0-setup",[86],{"type":43,"value":87},"Step 0 - Setup",{"type":38,"tag":89,"props":90,"children":91},"ol",{},[92,106],{"type":38,"tag":93,"props":94,"children":95},"li",{},[96,98,104],{"type":43,"value":97},"Install if missing: ",{"type":38,"tag":65,"props":99,"children":101},{"className":100},[],[102],{"type":43,"value":103},"pip install \"agentops-accelerator @ git+https:\u002F\u002Fgithub.com\u002FAzure\u002Fagentops.git@main\"",{"type":43,"value":105},".",{"type":38,"tag":93,"props":107,"children":108},{},[109,111,117,119,125,127,133,135,140,142,148,150,156,158,164,166,172,174,180,181,187,189,195],{"type":43,"value":110},"If ",{"type":38,"tag":65,"props":112,"children":114},{"className":113},[],[115],{"type":43,"value":116},"agentops.yaml",{"type":43,"value":118}," does not exist at the project root, run ",{"type":38,"tag":65,"props":120,"children":122},{"className":121},[],[123],{"type":43,"value":124},"agentops init",{"type":43,"value":126},".\nThe init wizard prompts (azd-style) for the Foundry project endpoint,\nagent reference, and dataset path, persists each answer to\n",{"type":38,"tag":65,"props":128,"children":130},{"className":129},[],[131],{"type":43,"value":132},".agentops\u002F.env",{"type":43,"value":134}," + ",{"type":38,"tag":65,"props":136,"children":138},{"className":137},[],[139],{"type":43,"value":116},{"type":43,"value":141}," as it goes. Existing azd workspaces, or\nruns with ",{"type":38,"tag":65,"props":143,"children":145},{"className":144},[],[146],{"type":43,"value":147},"--azd-env",{"type":43,"value":149},", use ",{"type":38,"tag":65,"props":151,"children":153},{"className":152},[],[154],{"type":43,"value":155},".azure\u002F\u003Cenv>\u002F.env",{"type":43,"value":157}," instead. Pass ",{"type":38,"tag":65,"props":159,"children":161},{"className":160},[],[162],{"type":43,"value":163},"--no-prompt",{"type":43,"value":165},"\nplus the explicit flags\n(",{"type":38,"tag":65,"props":167,"children":169},{"className":168},[],[170],{"type":43,"value":171},"--project-endpoint",{"type":43,"value":173},", ",{"type":38,"tag":65,"props":175,"children":177},{"className":176},[],[178],{"type":43,"value":179},"--agent",{"type":43,"value":173},{"type":38,"tag":65,"props":182,"children":184},{"className":183},[],[185],{"type":43,"value":186},"--dataset",{"type":43,"value":188},", …) for non-interactive\nruns. Run ",{"type":38,"tag":65,"props":190,"children":192},{"className":191},[],[193],{"type":43,"value":194},"agentops init show",{"type":43,"value":196}," later to inspect the resolved config.",{"type":38,"tag":82,"props":198,"children":200},{"id":199},"step-05-ensure-agent-build-and-data-plane-rbac-on-the-ai-services-account",[201],{"type":43,"value":202},"Step 0.5 - Ensure agent-build and data-plane RBAC on the AI Services account",{"type":38,"tag":46,"props":204,"children":205},{},[206,208,213,215,221,223,229,231,237,239,243,245,251,253,258,260,266],{"type":43,"value":207},"AgentOps eval (cloud graders ",{"type":38,"tag":57,"props":209,"children":210},{},[211],{"type":43,"value":212},"and",{"type":43,"value":214}," local AI-assisted evaluators) calls\n",{"type":38,"tag":65,"props":216,"children":218},{"className":217},[],[219],{"type":43,"value":220},"\u002Fopenai\u002Fdeployments\u002F...\u002Fchat\u002Fcompletions",{"type":43,"value":222}," on the AI Services account\nthat backs the Foundry project. Creating a project through the Foundry\nportal only assigns the user ",{"type":38,"tag":65,"props":224,"children":226},{"className":225},[],[227],{"type":43,"value":228},"Foundry User",{"type":43,"value":230}," at the ",{"type":38,"tag":232,"props":233,"children":234},"em",{},[235],{"type":43,"value":236},"project",{"type":43,"value":238}," scope,\nwhich does ",{"type":38,"tag":57,"props":240,"children":241},{},[242],{"type":43,"value":61},{"type":43,"value":244}," cover the parent account. The Foundry UI may then block\nagent creation with \"You don't have permission to build agents in this\nproject\" and ask for the old portal label, ",{"type":38,"tag":65,"props":246,"children":248},{"className":247},[],[249],{"type":43,"value":250},"Azure AI User",{"type":43,"value":252},"; the current Azure\nRBAC role is ",{"type":38,"tag":65,"props":254,"children":256},{"className":255},[],[257],{"type":43,"value":228},{"type":43,"value":259}," (",{"type":38,"tag":65,"props":261,"children":263},{"className":262},[],[264],{"type":43,"value":265},"53ca6127-db72-4b80-b1b0-d745d6d5456d",{"type":43,"value":267},").",{"type":38,"tag":46,"props":269,"children":270},{},[271,273,279,281,287,289,295,297,302,304,310,312,318,320,326],{"type":43,"value":272},"You also need ",{"type":38,"tag":65,"props":274,"children":276},{"className":275},[],[277],{"type":43,"value":278},"Cognitive Services OpenAI User",{"type":43,"value":280},"\n(",{"type":38,"tag":65,"props":282,"children":284},{"className":283},[],[285],{"type":43,"value":286},"5e0bd9bd-7b93-4f28-af87-19fc36ad61bd",{"type":43,"value":288},") for OpenAI data-plane actions on\nthe parent account. Subscription ",{"type":38,"tag":65,"props":290,"children":292},{"className":291},[],[293],{"type":43,"value":294},"Owner",{"type":43,"value":296}," is insufficient because the built-in\n",{"type":38,"tag":65,"props":298,"children":300},{"className":299},[],[301],{"type":43,"value":294},{"type":43,"value":303}," role has ",{"type":38,"tag":65,"props":305,"children":307},{"className":306},[],[308],{"type":43,"value":309},"actions: [\"*\"]",{"type":43,"value":311}," but ",{"type":38,"tag":65,"props":313,"children":315},{"className":314},[],[316],{"type":43,"value":317},"dataActions: []",{"type":43,"value":319},". The first\n",{"type":38,"tag":65,"props":321,"children":323},{"className":322},[],[324],{"type":43,"value":325},"agentops eval run",{"type":43,"value":327}," against a fresh workspace otherwise fails with:",{"type":38,"tag":329,"props":330,"children":334},"pre",{"className":331,"code":333,"language":43},[332],"language-text","PermissionDenied … lacks the required data action\n'Microsoft.CognitiveServices\u002Faccounts\u002FOpenAI\u002Fdeployments\u002Fchat\u002Fcompletions\u002Faction'\n",[335],{"type":38,"tag":65,"props":336,"children":338},{"__ignoreMap":337},"",[339],{"type":43,"value":333},{"type":38,"tag":46,"props":341,"children":342},{},[343,345,350,352,357,359,365,367,373],{"type":43,"value":344},"Run this preflight before Step 1. It must grant ",{"type":38,"tag":65,"props":346,"children":348},{"className":347},[],[349],{"type":43,"value":228},{"type":43,"value":351}," and\n",{"type":38,"tag":65,"props":353,"children":355},{"className":354},[],[356],{"type":43,"value":278},{"type":43,"value":358}," to the signed-in user on the AI Services\naccount, plus the OpenAI data-plane role to any Foundry\u002FAzure AI managed\nidentities in the resource group. Cloud evaluations run server-side and some\ngraders authenticate as those managed identities, so assigning only the user\ncan still produce intermittent ",{"type":38,"tag":65,"props":360,"children":362},{"className":361},[],[363],{"type":43,"value":364},"AuthenticationError",{"type":43,"value":366}," grader failures. The\ncommands are idempotent (",{"type":38,"tag":65,"props":368,"children":370},{"className":369},[],[371],{"type":43,"value":372},"RoleAssignmentExists",{"type":43,"value":374}," means the role was already\ngranted):",{"type":38,"tag":329,"props":376,"children":380},{"className":377,"code":378,"language":379,"meta":337,"style":337},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# 1. Resolve the AI Services account from agentops.yaml \u002F .azure\u002F\u003Cenv>\u002F.env\nPROJECT_ENDPOINT=$(grep -h '^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT' .azure\u002F*\u002F.env .agentops\u002F.env 2>\u002Fdev\u002Fnull | tail -1 | cut -d= -f2- | tr -d '\"')\nACCOUNT_HOST=$(echo \"$PROJECT_ENDPOINT\" | awk -F[\u002F:] '{print $4}')\nACCOUNT_NAME=$(echo \"$ACCOUNT_HOST\" | cut -d. -f1)\n\n# 2. Resolve subscription, resource group, account scope, and signed-in object ID\nSUB_ID=$(az account show --query id -o tsv)\nRG=$(az cognitiveservices account list --subscription \"$SUB_ID\" --query \"[?name=='$ACCOUNT_NAME'].resourceGroup | [0]\" -o tsv)\nACCOUNT_ID=$(az cognitiveservices account show -g \"$RG\" -n \"$ACCOUNT_NAME\" --query id -o tsv)\nOBJ_ID=$(az ad signed-in-user show --query id -o tsv)\n\n# 3. Grant the user portal build access and data-plane access at account scope.\naz role assignment create \\\n  --assignee \"$OBJ_ID\" \\\n  --role \"53ca6127-db72-4b80-b1b0-d745d6d5456d\" \\\n  --scope \"$ACCOUNT_ID\"\n\naz role assignment create \\\n  --assignee \"$OBJ_ID\" \\\n  --role \"5e0bd9bd-7b93-4f28-af87-19fc36ad61bd\" \\\n  --scope \"$ACCOUNT_ID\"\n\n# 4. Grant the same data-plane role to Foundry\u002FAzure AI managed identities.\naz resource list -g \"$RG\" \\\n  --query \"[?identity.principalId!=null].identity.principalId\" -o tsv |\nwhile read -r PRINCIPAL_ID; do\n  [ -z \"$PRINCIPAL_ID\" ] && continue\n  az role assignment create \\\n    --assignee-object-id \"$PRINCIPAL_ID\" \\\n    --assignee-principal-type ServicePrincipal \\\n    --role \"5e0bd9bd-7b93-4f28-af87-19fc36ad61bd\" \\\n    --scope \"$ACCOUNT_ID\"\ndone\n","bash",[381],{"type":38,"tag":65,"props":382,"children":383},{"__ignoreMap":337},[384,396,535,614,666,676,685,736,824,908,958,966,975,1003,1029,1054,1077,1085,1109,1133,1157,1177,1185,1194,1231,1266,1301,1343,1368,1393,1411,1436,1457],{"type":38,"tag":385,"props":386,"children":389},"span",{"class":387,"line":388},"line",1,[390],{"type":38,"tag":385,"props":391,"children":393},{"style":392},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[394],{"type":43,"value":395},"# 1. Resolve the AI Services account from agentops.yaml \u002F .azure\u002F\u003Cenv>\u002F.env\n",{"type":38,"tag":385,"props":397,"children":399},{"class":387,"line":398},2,[400,406,412,418,424,429,434,439,444,449,454,459,464,469,474,479,484,488,493,498,503,507,512,517,521,526,530],{"type":38,"tag":385,"props":401,"children":403},{"style":402},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[404],{"type":43,"value":405},"PROJECT_ENDPOINT",{"type":38,"tag":385,"props":407,"children":409},{"style":408},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[410],{"type":43,"value":411},"=$(",{"type":38,"tag":385,"props":413,"children":415},{"style":414},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[416],{"type":43,"value":417},"grep",{"type":38,"tag":385,"props":419,"children":421},{"style":420},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[422],{"type":43,"value":423}," -h",{"type":38,"tag":385,"props":425,"children":426},{"style":408},[427],{"type":43,"value":428}," '",{"type":38,"tag":385,"props":430,"children":431},{"style":420},[432],{"type":43,"value":433},"^AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",{"type":38,"tag":385,"props":435,"children":436},{"style":408},[437],{"type":43,"value":438},"'",{"type":38,"tag":385,"props":440,"children":441},{"style":420},[442],{"type":43,"value":443}," .azure\u002F",{"type":38,"tag":385,"props":445,"children":446},{"style":402},[447],{"type":43,"value":448},"*",{"type":38,"tag":385,"props":450,"children":451},{"style":420},[452],{"type":43,"value":453},"\u002F.env",{"type":38,"tag":385,"props":455,"children":456},{"style":420},[457],{"type":43,"value":458}," .agentops\u002F.env",{"type":38,"tag":385,"props":460,"children":461},{"style":408},[462],{"type":43,"value":463}," 2>",{"type":38,"tag":385,"props":465,"children":466},{"style":420},[467],{"type":43,"value":468},"\u002Fdev\u002Fnull",{"type":38,"tag":385,"props":470,"children":471},{"style":408},[472],{"type":43,"value":473}," |",{"type":38,"tag":385,"props":475,"children":476},{"style":414},[477],{"type":43,"value":478}," tail",{"type":38,"tag":385,"props":480,"children":481},{"style":420},[482],{"type":43,"value":483}," -1",{"type":38,"tag":385,"props":485,"children":486},{"style":408},[487],{"type":43,"value":473},{"type":38,"tag":385,"props":489,"children":490},{"style":414},[491],{"type":43,"value":492}," cut",{"type":38,"tag":385,"props":494,"children":495},{"style":420},[496],{"type":43,"value":497}," -d=",{"type":38,"tag":385,"props":499,"children":500},{"style":420},[501],{"type":43,"value":502}," -f2-",{"type":38,"tag":385,"props":504,"children":505},{"style":408},[506],{"type":43,"value":473},{"type":38,"tag":385,"props":508,"children":509},{"style":414},[510],{"type":43,"value":511}," tr",{"type":38,"tag":385,"props":513,"children":514},{"style":420},[515],{"type":43,"value":516}," -d",{"type":38,"tag":385,"props":518,"children":519},{"style":408},[520],{"type":43,"value":428},{"type":38,"tag":385,"props":522,"children":523},{"style":420},[524],{"type":43,"value":525},"\"",{"type":38,"tag":385,"props":527,"children":528},{"style":408},[529],{"type":43,"value":438},{"type":38,"tag":385,"props":531,"children":532},{"style":408},[533],{"type":43,"value":534},")\n",{"type":38,"tag":385,"props":536,"children":538},{"class":387,"line":537},3,[539,544,548,554,559,564,568,572,577,582,587,592,597,601,606,610],{"type":38,"tag":385,"props":540,"children":541},{"style":402},[542],{"type":43,"value":543},"ACCOUNT_HOST",{"type":38,"tag":385,"props":545,"children":546},{"style":408},[547],{"type":43,"value":411},{"type":38,"tag":385,"props":549,"children":551},{"style":550},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[552],{"type":43,"value":553},"echo",{"type":38,"tag":385,"props":555,"children":556},{"style":408},[557],{"type":43,"value":558}," \"",{"type":38,"tag":385,"props":560,"children":561},{"style":402},[562],{"type":43,"value":563},"$PROJECT_ENDPOINT",{"type":38,"tag":385,"props":565,"children":566},{"style":408},[567],{"type":43,"value":525},{"type":38,"tag":385,"props":569,"children":570},{"style":408},[571],{"type":43,"value":473},{"type":38,"tag":385,"props":573,"children":574},{"style":414},[575],{"type":43,"value":576}," awk",{"type":38,"tag":385,"props":578,"children":579},{"style":420},[580],{"type":43,"value":581}," -F",{"type":38,"tag":385,"props":583,"children":584},{"style":408},[585],{"type":43,"value":586},"[",{"type":38,"tag":385,"props":588,"children":589},{"style":420},[590],{"type":43,"value":591},"\u002F:",{"type":38,"tag":385,"props":593,"children":594},{"style":408},[595],{"type":43,"value":596},"]",{"type":38,"tag":385,"props":598,"children":599},{"style":408},[600],{"type":43,"value":428},{"type":38,"tag":385,"props":602,"children":603},{"style":420},[604],{"type":43,"value":605},"{print $4}",{"type":38,"tag":385,"props":607,"children":608},{"style":408},[609],{"type":43,"value":438},{"type":38,"tag":385,"props":611,"children":612},{"style":408},[613],{"type":43,"value":534},{"type":38,"tag":385,"props":615,"children":617},{"class":387,"line":616},4,[618,623,627,631,635,640,644,648,652,657,662],{"type":38,"tag":385,"props":619,"children":620},{"style":402},[621],{"type":43,"value":622},"ACCOUNT_NAME",{"type":38,"tag":385,"props":624,"children":625},{"style":408},[626],{"type":43,"value":411},{"type":38,"tag":385,"props":628,"children":629},{"style":550},[630],{"type":43,"value":553},{"type":38,"tag":385,"props":632,"children":633},{"style":408},[634],{"type":43,"value":558},{"type":38,"tag":385,"props":636,"children":637},{"style":402},[638],{"type":43,"value":639},"$ACCOUNT_HOST",{"type":38,"tag":385,"props":641,"children":642},{"style":408},[643],{"type":43,"value":525},{"type":38,"tag":385,"props":645,"children":646},{"style":408},[647],{"type":43,"value":473},{"type":38,"tag":385,"props":649,"children":650},{"style":414},[651],{"type":43,"value":492},{"type":38,"tag":385,"props":653,"children":654},{"style":420},[655],{"type":43,"value":656}," -d.",{"type":38,"tag":385,"props":658,"children":659},{"style":420},[660],{"type":43,"value":661}," -f1",{"type":38,"tag":385,"props":663,"children":664},{"style":408},[665],{"type":43,"value":534},{"type":38,"tag":385,"props":667,"children":669},{"class":387,"line":668},5,[670],{"type":38,"tag":385,"props":671,"children":673},{"emptyLinePlaceholder":672},true,[674],{"type":43,"value":675},"\n",{"type":38,"tag":385,"props":677,"children":679},{"class":387,"line":678},6,[680],{"type":38,"tag":385,"props":681,"children":682},{"style":392},[683],{"type":43,"value":684},"# 2. Resolve subscription, resource group, account scope, and signed-in object ID\n",{"type":38,"tag":385,"props":686,"children":687},{"class":387,"line":25},[688,693,697,702,707,712,717,722,727,732],{"type":38,"tag":385,"props":689,"children":690},{"style":402},[691],{"type":43,"value":692},"SUB_ID",{"type":38,"tag":385,"props":694,"children":695},{"style":408},[696],{"type":43,"value":411},{"type":38,"tag":385,"props":698,"children":699},{"style":414},[700],{"type":43,"value":701},"az",{"type":38,"tag":385,"props":703,"children":704},{"style":420},[705],{"type":43,"value":706}," account",{"type":38,"tag":385,"props":708,"children":709},{"style":420},[710],{"type":43,"value":711}," show",{"type":38,"tag":385,"props":713,"children":714},{"style":420},[715],{"type":43,"value":716}," --query",{"type":38,"tag":385,"props":718,"children":719},{"style":420},[720],{"type":43,"value":721}," id",{"type":38,"tag":385,"props":723,"children":724},{"style":420},[725],{"type":43,"value":726}," -o",{"type":38,"tag":385,"props":728,"children":729},{"style":420},[730],{"type":43,"value":731}," tsv",{"type":38,"tag":385,"props":733,"children":734},{"style":408},[735],{"type":43,"value":534},{"type":38,"tag":385,"props":737,"children":739},{"class":387,"line":738},8,[740,745,749,753,758,762,767,772,776,781,785,789,793,798,803,808,812,816,820],{"type":38,"tag":385,"props":741,"children":742},{"style":402},[743],{"type":43,"value":744},"RG",{"type":38,"tag":385,"props":746,"children":747},{"style":408},[748],{"type":43,"value":411},{"type":38,"tag":385,"props":750,"children":751},{"style":414},[752],{"type":43,"value":701},{"type":38,"tag":385,"props":754,"children":755},{"style":420},[756],{"type":43,"value":757}," cognitiveservices",{"type":38,"tag":385,"props":759,"children":760},{"style":420},[761],{"type":43,"value":706},{"type":38,"tag":385,"props":763,"children":764},{"style":420},[765],{"type":43,"value":766}," list",{"type":38,"tag":385,"props":768,"children":769},{"style":420},[770],{"type":43,"value":771}," --subscription",{"type":38,"tag":385,"props":773,"children":774},{"style":408},[775],{"type":43,"value":558},{"type":38,"tag":385,"props":777,"children":778},{"style":402},[779],{"type":43,"value":780},"$SUB_ID",{"type":38,"tag":385,"props":782,"children":783},{"style":408},[784],{"type":43,"value":525},{"type":38,"tag":385,"props":786,"children":787},{"style":420},[788],{"type":43,"value":716},{"type":38,"tag":385,"props":790,"children":791},{"style":408},[792],{"type":43,"value":558},{"type":38,"tag":385,"props":794,"children":795},{"style":420},[796],{"type":43,"value":797},"[?name=='",{"type":38,"tag":385,"props":799,"children":800},{"style":402},[801],{"type":43,"value":802},"$ACCOUNT_NAME",{"type":38,"tag":385,"props":804,"children":805},{"style":420},[806],{"type":43,"value":807},"'].resourceGroup | [0]",{"type":38,"tag":385,"props":809,"children":810},{"style":408},[811],{"type":43,"value":525},{"type":38,"tag":385,"props":813,"children":814},{"style":420},[815],{"type":43,"value":726},{"type":38,"tag":385,"props":817,"children":818},{"style":420},[819],{"type":43,"value":731},{"type":38,"tag":385,"props":821,"children":822},{"style":408},[823],{"type":43,"value":534},{"type":38,"tag":385,"props":825,"children":827},{"class":387,"line":826},9,[828,833,837,841,845,849,853,858,862,867,871,876,880,884,888,892,896,900,904],{"type":38,"tag":385,"props":829,"children":830},{"style":402},[831],{"type":43,"value":832},"ACCOUNT_ID",{"type":38,"tag":385,"props":834,"children":835},{"style":408},[836],{"type":43,"value":411},{"type":38,"tag":385,"props":838,"children":839},{"style":414},[840],{"type":43,"value":701},{"type":38,"tag":385,"props":842,"children":843},{"style":420},[844],{"type":43,"value":757},{"type":38,"tag":385,"props":846,"children":847},{"style":420},[848],{"type":43,"value":706},{"type":38,"tag":385,"props":850,"children":851},{"style":420},[852],{"type":43,"value":711},{"type":38,"tag":385,"props":854,"children":855},{"style":420},[856],{"type":43,"value":857}," -g",{"type":38,"tag":385,"props":859,"children":860},{"style":408},[861],{"type":43,"value":558},{"type":38,"tag":385,"props":863,"children":864},{"style":402},[865],{"type":43,"value":866},"$RG",{"type":38,"tag":385,"props":868,"children":869},{"style":408},[870],{"type":43,"value":525},{"type":38,"tag":385,"props":872,"children":873},{"style":420},[874],{"type":43,"value":875}," -n",{"type":38,"tag":385,"props":877,"children":878},{"style":408},[879],{"type":43,"value":558},{"type":38,"tag":385,"props":881,"children":882},{"style":402},[883],{"type":43,"value":802},{"type":38,"tag":385,"props":885,"children":886},{"style":408},[887],{"type":43,"value":525},{"type":38,"tag":385,"props":889,"children":890},{"style":420},[891],{"type":43,"value":716},{"type":38,"tag":385,"props":893,"children":894},{"style":420},[895],{"type":43,"value":721},{"type":38,"tag":385,"props":897,"children":898},{"style":420},[899],{"type":43,"value":726},{"type":38,"tag":385,"props":901,"children":902},{"style":420},[903],{"type":43,"value":731},{"type":38,"tag":385,"props":905,"children":906},{"style":408},[907],{"type":43,"value":534},{"type":38,"tag":385,"props":909,"children":910},{"class":387,"line":21},[911,916,920,924,929,934,938,942,946,950,954],{"type":38,"tag":385,"props":912,"children":913},{"style":402},[914],{"type":43,"value":915},"OBJ_ID",{"type":38,"tag":385,"props":917,"children":918},{"style":408},[919],{"type":43,"value":411},{"type":38,"tag":385,"props":921,"children":922},{"style":414},[923],{"type":43,"value":701},{"type":38,"tag":385,"props":925,"children":926},{"style":420},[927],{"type":43,"value":928}," ad",{"type":38,"tag":385,"props":930,"children":931},{"style":420},[932],{"type":43,"value":933}," signed-in-user",{"type":38,"tag":385,"props":935,"children":936},{"style":420},[937],{"type":43,"value":711},{"type":38,"tag":385,"props":939,"children":940},{"style":420},[941],{"type":43,"value":716},{"type":38,"tag":385,"props":943,"children":944},{"style":420},[945],{"type":43,"value":721},{"type":38,"tag":385,"props":947,"children":948},{"style":420},[949],{"type":43,"value":726},{"type":38,"tag":385,"props":951,"children":952},{"style":420},[953],{"type":43,"value":731},{"type":38,"tag":385,"props":955,"children":956},{"style":408},[957],{"type":43,"value":534},{"type":38,"tag":385,"props":959,"children":961},{"class":387,"line":960},11,[962],{"type":38,"tag":385,"props":963,"children":964},{"emptyLinePlaceholder":672},[965],{"type":43,"value":675},{"type":38,"tag":385,"props":967,"children":969},{"class":387,"line":968},12,[970],{"type":38,"tag":385,"props":971,"children":972},{"style":392},[973],{"type":43,"value":974},"# 3. Grant the user portal build access and data-plane access at account scope.\n",{"type":38,"tag":385,"props":976,"children":978},{"class":387,"line":977},13,[979,983,988,993,998],{"type":38,"tag":385,"props":980,"children":981},{"style":414},[982],{"type":43,"value":701},{"type":38,"tag":385,"props":984,"children":985},{"style":420},[986],{"type":43,"value":987}," role",{"type":38,"tag":385,"props":989,"children":990},{"style":420},[991],{"type":43,"value":992}," assignment",{"type":38,"tag":385,"props":994,"children":995},{"style":420},[996],{"type":43,"value":997}," create",{"type":38,"tag":385,"props":999,"children":1000},{"style":402},[1001],{"type":43,"value":1002}," \\\n",{"type":38,"tag":385,"props":1004,"children":1006},{"class":387,"line":1005},14,[1007,1012,1016,1021,1025],{"type":38,"tag":385,"props":1008,"children":1009},{"style":420},[1010],{"type":43,"value":1011},"  --assignee",{"type":38,"tag":385,"props":1013,"children":1014},{"style":408},[1015],{"type":43,"value":558},{"type":38,"tag":385,"props":1017,"children":1018},{"style":402},[1019],{"type":43,"value":1020},"$OBJ_ID",{"type":38,"tag":385,"props":1022,"children":1023},{"style":408},[1024],{"type":43,"value":525},{"type":38,"tag":385,"props":1026,"children":1027},{"style":402},[1028],{"type":43,"value":1002},{"type":38,"tag":385,"props":1030,"children":1032},{"class":387,"line":1031},15,[1033,1038,1042,1046,1050],{"type":38,"tag":385,"props":1034,"children":1035},{"style":420},[1036],{"type":43,"value":1037},"  --role",{"type":38,"tag":385,"props":1039,"children":1040},{"style":408},[1041],{"type":43,"value":558},{"type":38,"tag":385,"props":1043,"children":1044},{"style":420},[1045],{"type":43,"value":265},{"type":38,"tag":385,"props":1047,"children":1048},{"style":408},[1049],{"type":43,"value":525},{"type":38,"tag":385,"props":1051,"children":1052},{"style":402},[1053],{"type":43,"value":1002},{"type":38,"tag":385,"props":1055,"children":1057},{"class":387,"line":1056},16,[1058,1063,1067,1072],{"type":38,"tag":385,"props":1059,"children":1060},{"style":420},[1061],{"type":43,"value":1062},"  --scope",{"type":38,"tag":385,"props":1064,"children":1065},{"style":408},[1066],{"type":43,"value":558},{"type":38,"tag":385,"props":1068,"children":1069},{"style":402},[1070],{"type":43,"value":1071},"$ACCOUNT_ID",{"type":38,"tag":385,"props":1073,"children":1074},{"style":408},[1075],{"type":43,"value":1076},"\"\n",{"type":38,"tag":385,"props":1078,"children":1080},{"class":387,"line":1079},17,[1081],{"type":38,"tag":385,"props":1082,"children":1083},{"emptyLinePlaceholder":672},[1084],{"type":43,"value":675},{"type":38,"tag":385,"props":1086,"children":1088},{"class":387,"line":1087},18,[1089,1093,1097,1101,1105],{"type":38,"tag":385,"props":1090,"children":1091},{"style":414},[1092],{"type":43,"value":701},{"type":38,"tag":385,"props":1094,"children":1095},{"style":420},[1096],{"type":43,"value":987},{"type":38,"tag":385,"props":1098,"children":1099},{"style":420},[1100],{"type":43,"value":992},{"type":38,"tag":385,"props":1102,"children":1103},{"style":420},[1104],{"type":43,"value":997},{"type":38,"tag":385,"props":1106,"children":1107},{"style":402},[1108],{"type":43,"value":1002},{"type":38,"tag":385,"props":1110,"children":1112},{"class":387,"line":1111},19,[1113,1117,1121,1125,1129],{"type":38,"tag":385,"props":1114,"children":1115},{"style":420},[1116],{"type":43,"value":1011},{"type":38,"tag":385,"props":1118,"children":1119},{"style":408},[1120],{"type":43,"value":558},{"type":38,"tag":385,"props":1122,"children":1123},{"style":402},[1124],{"type":43,"value":1020},{"type":38,"tag":385,"props":1126,"children":1127},{"style":408},[1128],{"type":43,"value":525},{"type":38,"tag":385,"props":1130,"children":1131},{"style":402},[1132],{"type":43,"value":1002},{"type":38,"tag":385,"props":1134,"children":1136},{"class":387,"line":1135},20,[1137,1141,1145,1149,1153],{"type":38,"tag":385,"props":1138,"children":1139},{"style":420},[1140],{"type":43,"value":1037},{"type":38,"tag":385,"props":1142,"children":1143},{"style":408},[1144],{"type":43,"value":558},{"type":38,"tag":385,"props":1146,"children":1147},{"style":420},[1148],{"type":43,"value":286},{"type":38,"tag":385,"props":1150,"children":1151},{"style":408},[1152],{"type":43,"value":525},{"type":38,"tag":385,"props":1154,"children":1155},{"style":402},[1156],{"type":43,"value":1002},{"type":38,"tag":385,"props":1158,"children":1160},{"class":387,"line":1159},21,[1161,1165,1169,1173],{"type":38,"tag":385,"props":1162,"children":1163},{"style":420},[1164],{"type":43,"value":1062},{"type":38,"tag":385,"props":1166,"children":1167},{"style":408},[1168],{"type":43,"value":558},{"type":38,"tag":385,"props":1170,"children":1171},{"style":402},[1172],{"type":43,"value":1071},{"type":38,"tag":385,"props":1174,"children":1175},{"style":408},[1176],{"type":43,"value":1076},{"type":38,"tag":385,"props":1178,"children":1180},{"class":387,"line":1179},22,[1181],{"type":38,"tag":385,"props":1182,"children":1183},{"emptyLinePlaceholder":672},[1184],{"type":43,"value":675},{"type":38,"tag":385,"props":1186,"children":1188},{"class":387,"line":1187},23,[1189],{"type":38,"tag":385,"props":1190,"children":1191},{"style":392},[1192],{"type":43,"value":1193},"# 4. Grant the same data-plane role to Foundry\u002FAzure AI managed identities.\n",{"type":38,"tag":385,"props":1195,"children":1197},{"class":387,"line":1196},24,[1198,1202,1207,1211,1215,1219,1223,1227],{"type":38,"tag":385,"props":1199,"children":1200},{"style":414},[1201],{"type":43,"value":701},{"type":38,"tag":385,"props":1203,"children":1204},{"style":420},[1205],{"type":43,"value":1206}," resource",{"type":38,"tag":385,"props":1208,"children":1209},{"style":420},[1210],{"type":43,"value":766},{"type":38,"tag":385,"props":1212,"children":1213},{"style":420},[1214],{"type":43,"value":857},{"type":38,"tag":385,"props":1216,"children":1217},{"style":408},[1218],{"type":43,"value":558},{"type":38,"tag":385,"props":1220,"children":1221},{"style":402},[1222],{"type":43,"value":866},{"type":38,"tag":385,"props":1224,"children":1225},{"style":408},[1226],{"type":43,"value":525},{"type":38,"tag":385,"props":1228,"children":1229},{"style":402},[1230],{"type":43,"value":1002},{"type":38,"tag":385,"props":1232,"children":1234},{"class":387,"line":1233},25,[1235,1240,1244,1249,1253,1257,1261],{"type":38,"tag":385,"props":1236,"children":1237},{"style":420},[1238],{"type":43,"value":1239},"  --query",{"type":38,"tag":385,"props":1241,"children":1242},{"style":408},[1243],{"type":43,"value":558},{"type":38,"tag":385,"props":1245,"children":1246},{"style":420},[1247],{"type":43,"value":1248},"[?identity.principalId!=null].identity.principalId",{"type":38,"tag":385,"props":1250,"children":1251},{"style":408},[1252],{"type":43,"value":525},{"type":38,"tag":385,"props":1254,"children":1255},{"style":420},[1256],{"type":43,"value":726},{"type":38,"tag":385,"props":1258,"children":1259},{"style":420},[1260],{"type":43,"value":731},{"type":38,"tag":385,"props":1262,"children":1263},{"style":408},[1264],{"type":43,"value":1265}," |\n",{"type":38,"tag":385,"props":1267,"children":1269},{"class":387,"line":1268},26,[1270,1276,1281,1286,1291,1296],{"type":38,"tag":385,"props":1271,"children":1273},{"style":1272},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[1274],{"type":43,"value":1275},"while",{"type":38,"tag":385,"props":1277,"children":1278},{"style":420},[1279],{"type":43,"value":1280}," read",{"type":38,"tag":385,"props":1282,"children":1283},{"style":420},[1284],{"type":43,"value":1285}," -r",{"type":38,"tag":385,"props":1287,"children":1288},{"style":420},[1289],{"type":43,"value":1290}," PRINCIPAL_ID",{"type":38,"tag":385,"props":1292,"children":1293},{"style":408},[1294],{"type":43,"value":1295},";",{"type":38,"tag":385,"props":1297,"children":1298},{"style":1272},[1299],{"type":43,"value":1300}," do\n",{"type":38,"tag":385,"props":1302,"children":1304},{"class":387,"line":1303},27,[1305,1310,1315,1319,1324,1328,1333,1338],{"type":38,"tag":385,"props":1306,"children":1307},{"style":408},[1308],{"type":43,"value":1309},"  [",{"type":38,"tag":385,"props":1311,"children":1312},{"style":408},[1313],{"type":43,"value":1314}," -z",{"type":38,"tag":385,"props":1316,"children":1317},{"style":408},[1318],{"type":43,"value":558},{"type":38,"tag":385,"props":1320,"children":1321},{"style":402},[1322],{"type":43,"value":1323},"$PRINCIPAL_ID",{"type":38,"tag":385,"props":1325,"children":1326},{"style":408},[1327],{"type":43,"value":525},{"type":38,"tag":385,"props":1329,"children":1330},{"style":408},[1331],{"type":43,"value":1332}," ]",{"type":38,"tag":385,"props":1334,"children":1335},{"style":408},[1336],{"type":43,"value":1337}," &&",{"type":38,"tag":385,"props":1339,"children":1340},{"style":1272},[1341],{"type":43,"value":1342}," continue\n",{"type":38,"tag":385,"props":1344,"children":1346},{"class":387,"line":1345},28,[1347,1352,1356,1360,1364],{"type":38,"tag":385,"props":1348,"children":1349},{"style":414},[1350],{"type":43,"value":1351},"  az",{"type":38,"tag":385,"props":1353,"children":1354},{"style":420},[1355],{"type":43,"value":987},{"type":38,"tag":385,"props":1357,"children":1358},{"style":420},[1359],{"type":43,"value":992},{"type":38,"tag":385,"props":1361,"children":1362},{"style":420},[1363],{"type":43,"value":997},{"type":38,"tag":385,"props":1365,"children":1366},{"style":402},[1367],{"type":43,"value":1002},{"type":38,"tag":385,"props":1369,"children":1371},{"class":387,"line":1370},29,[1372,1377,1381,1385,1389],{"type":38,"tag":385,"props":1373,"children":1374},{"style":420},[1375],{"type":43,"value":1376},"    --assignee-object-id",{"type":38,"tag":385,"props":1378,"children":1379},{"style":408},[1380],{"type":43,"value":558},{"type":38,"tag":385,"props":1382,"children":1383},{"style":402},[1384],{"type":43,"value":1323},{"type":38,"tag":385,"props":1386,"children":1387},{"style":408},[1388],{"type":43,"value":525},{"type":38,"tag":385,"props":1390,"children":1391},{"style":402},[1392],{"type":43,"value":1002},{"type":38,"tag":385,"props":1394,"children":1396},{"class":387,"line":1395},30,[1397,1402,1407],{"type":38,"tag":385,"props":1398,"children":1399},{"style":420},[1400],{"type":43,"value":1401},"    --assignee-principal-type",{"type":38,"tag":385,"props":1403,"children":1404},{"style":420},[1405],{"type":43,"value":1406}," ServicePrincipal",{"type":38,"tag":385,"props":1408,"children":1409},{"style":402},[1410],{"type":43,"value":1002},{"type":38,"tag":385,"props":1412,"children":1414},{"class":387,"line":1413},31,[1415,1420,1424,1428,1432],{"type":38,"tag":385,"props":1416,"children":1417},{"style":420},[1418],{"type":43,"value":1419},"    --role",{"type":38,"tag":385,"props":1421,"children":1422},{"style":408},[1423],{"type":43,"value":558},{"type":38,"tag":385,"props":1425,"children":1426},{"style":420},[1427],{"type":43,"value":286},{"type":38,"tag":385,"props":1429,"children":1430},{"style":408},[1431],{"type":43,"value":525},{"type":38,"tag":385,"props":1433,"children":1434},{"style":402},[1435],{"type":43,"value":1002},{"type":38,"tag":385,"props":1437,"children":1439},{"class":387,"line":1438},32,[1440,1445,1449,1453],{"type":38,"tag":385,"props":1441,"children":1442},{"style":420},[1443],{"type":43,"value":1444},"    --scope",{"type":38,"tag":385,"props":1446,"children":1447},{"style":408},[1448],{"type":43,"value":558},{"type":38,"tag":385,"props":1450,"children":1451},{"style":402},[1452],{"type":43,"value":1071},{"type":38,"tag":385,"props":1454,"children":1455},{"style":408},[1456],{"type":43,"value":1076},{"type":38,"tag":385,"props":1458,"children":1460},{"class":387,"line":1459},33,[1461],{"type":38,"tag":385,"props":1462,"children":1463},{"style":1272},[1464],{"type":43,"value":1465},"done\n",{"type":38,"tag":46,"props":1467,"children":1468},{},[1469,1471,1477,1479,1485],{"type":43,"value":1470},"PowerShell equivalent: replace ",{"type":38,"tag":65,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":43,"value":1476},"$(...)",{"type":43,"value":1478}," with the PowerShell variable\nassignments shown in ",{"type":38,"tag":65,"props":1480,"children":1482},{"className":1481},[],[1483],{"type":43,"value":1484},"docs\u002Ftutorial-prompt-agent.md",{"type":43,"value":105},{"type":38,"tag":46,"props":1487,"children":1488},{},[1489,1491,1497,1499,1505],{"type":43,"value":1490},"If the user has not run ",{"type":38,"tag":65,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":43,"value":1496},"az login",{"type":43,"value":1498}," yet, do that first. If\n",{"type":38,"tag":65,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":43,"value":1504},"az cognitiveservices account list",{"type":43,"value":1506}," returns an empty RG, the AI Services\naccount lives in a different subscription - ask the user which one.",{"type":38,"tag":46,"props":1508,"children":1509},{},[1510,1512,1517],{"type":43,"value":1511},"Skip this step only if the user explicitly says both roles are already\nassigned on the parent AI Services account, or if the user can already build\nagents in the Foundry UI and a previous ",{"type":38,"tag":65,"props":1513,"children":1515},{"className":1514},[],[1516],{"type":43,"value":325},{"type":43,"value":1518}," succeeded against\nthe same Foundry account.",{"type":38,"tag":46,"props":1520,"children":1521},{},[1522,1527,1529,1534,1536,1542,1544,1549,1551,1556],{"type":38,"tag":57,"props":1523,"children":1524},{},[1525],{"type":43,"value":1526},"Propagation:",{"type":43,"value":1528}," data-plane role assignments do not take effect\ninstantly — allow several minutes (occasionally up to ~15) before the\nfirst eval. The cloud\u002Flocal graders authenticate per call, so if the\nuser runs an eval immediately after this preflight and sees intermittent\n",{"type":38,"tag":65,"props":1530,"children":1532},{"className":1531},[],[1533],{"type":43,"value":364},{"type":43,"value":1535}," on a subset of graders plus\n",{"type":38,"tag":65,"props":1537,"children":1539},{"className":1538},[],[1540],{"type":43,"value":1541},"Threshold status: FAILED",{"type":43,"value":1543}," while the visible thresholds are green, that\nis propagation lag (a grader ",{"type":38,"tag":57,"props":1545,"children":1546},{},[1547],{"type":43,"value":1548},"execution",{"type":43,"value":1550}," failure), not a quality\nregression. Tell the user to wait a few minutes and re-run\n",{"type":38,"tag":65,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":43,"value":325},{"type":43,"value":1557},"; do not treat it as a failing gate or start changing\nthresholds.",{"type":38,"tag":82,"props":1559,"children":1561},{"id":1560},"step-1-analyze-evaluation-setup",[1562],{"type":43,"value":1563},"Step 1 - Analyze evaluation setup",{"type":38,"tag":46,"props":1565,"children":1566},{},[1567],{"type":43,"value":1568},"Run the deterministic local triage first:",{"type":38,"tag":329,"props":1570,"children":1572},{"className":377,"code":1571,"language":379,"meta":337,"style":337},"agentops eval analyze\n",[1573],{"type":38,"tag":65,"props":1574,"children":1575},{"__ignoreMap":337},[1576],{"type":38,"tag":385,"props":1577,"children":1578},{"class":387,"line":388},[1579,1584,1589],{"type":38,"tag":385,"props":1580,"children":1581},{"style":414},[1582],{"type":43,"value":1583},"agentops",{"type":38,"tag":385,"props":1585,"children":1586},{"style":420},[1587],{"type":43,"value":1588}," eval",{"type":38,"tag":385,"props":1590,"children":1591},{"style":420},[1592],{"type":43,"value":1593}," analyze\n",{"type":38,"tag":46,"props":1595,"children":1596},{},[1597,1599,1604,1606,1612,1614,1620,1622,1627],{"type":43,"value":1598},"Use its output to decide whether the repo is ready for ",{"type":38,"tag":65,"props":1600,"children":1602},{"className":1601},[],[1603],{"type":43,"value":325},{"type":43,"value":1605}," or\nneeds skill-assisted setup. If it recommends ",{"type":38,"tag":65,"props":1607,"children":1609},{"className":1608},[],[1610],{"type":43,"value":1611},"agentops-config",{"type":43,"value":1613},", fix the target\nand protocol. If it recommends ",{"type":38,"tag":65,"props":1615,"children":1617},{"className":1616},[],[1618],{"type":43,"value":1619},"agentops-dataset",{"type":43,"value":1621},", create\u002Fmap realistic JSONL\nrows. If it recommends ",{"type":38,"tag":65,"props":1623,"children":1625},{"className":1624},[],[1626],{"type":43,"value":4},{"type":43,"value":1628},", inspect the app scenario and evaluator\nexpectations before running.",{"type":38,"tag":82,"props":1630,"children":1632},{"id":1631},"step-2-identify-the-agent-target",[1633],{"type":43,"value":1634},"Step 2 - Identify the agent target",{"type":38,"tag":46,"props":1636,"children":1637},{},[1638,1640,1646,1648,1653],{"type":43,"value":1639},"Read the codebase (README, entry point, env vars) and pick the right value\nfor the ",{"type":38,"tag":65,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":43,"value":1645},"agent:",{"type":43,"value":1647}," field of ",{"type":38,"tag":65,"props":1649,"children":1651},{"className":1650},[],[1652],{"type":43,"value":116},{"type":43,"value":1654},":",{"type":38,"tag":1656,"props":1657,"children":1658},"table",{},[1659,1683],{"type":38,"tag":1660,"props":1661,"children":1662},"thead",{},[1663],{"type":38,"tag":1664,"props":1665,"children":1666},"tr",{},[1667,1673],{"type":38,"tag":1668,"props":1669,"children":1670},"th",{},[1671],{"type":43,"value":1672},"Pattern in code \u002F env",{"type":38,"tag":1668,"props":1674,"children":1675},{},[1676,1681],{"type":38,"tag":65,"props":1677,"children":1679},{"className":1678},[],[1680],{"type":43,"value":1645},{"type":43,"value":1682}," value",{"type":38,"tag":1684,"props":1685,"children":1686},"tbody",{},[1687,1711,1734,1751],{"type":38,"tag":1664,"props":1688,"children":1689},{},[1690,1702],{"type":38,"tag":1691,"props":1692,"children":1693},"td",{},[1694,1696],{"type":43,"value":1695},"Foundry Prompt Agent ID like ",{"type":38,"tag":65,"props":1697,"children":1699},{"className":1698},[],[1700],{"type":43,"value":1701},"name:1",{"type":38,"tag":1691,"props":1703,"children":1704},{},[1705],{"type":38,"tag":65,"props":1706,"children":1708},{"className":1707},[],[1709],{"type":43,"value":1710},"\"\u003Cname>:\u003Cversion>\"",{"type":38,"tag":1664,"props":1712,"children":1713},{},[1714,1725],{"type":38,"tag":1691,"props":1715,"children":1716},{},[1717,1719],{"type":43,"value":1718},"Foundry Hosted Agent endpoint URL ending in ",{"type":38,"tag":65,"props":1720,"children":1722},{"className":1721},[],[1723],{"type":43,"value":1724},"\u002Fagents\u002F...",{"type":38,"tag":1691,"props":1726,"children":1727},{},[1728],{"type":38,"tag":65,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":43,"value":1733},"\"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\u002Fagents\u002F...\"",{"type":38,"tag":1664,"props":1735,"children":1736},{},[1737,1742],{"type":38,"tag":1691,"props":1738,"children":1739},{},[1740],{"type":43,"value":1741},"Plain HTTP\u002FJSON endpoint (FastAPI, Express, ACA, AKS)",{"type":38,"tag":1691,"props":1743,"children":1744},{},[1745],{"type":38,"tag":65,"props":1746,"children":1748},{"className":1747},[],[1749],{"type":43,"value":1750},"\"https:\u002F\u002F\u003Chost>\u002F\u003Cpath>\"",{"type":38,"tag":1664,"props":1752,"children":1753},{},[1754,1759],{"type":38,"tag":1691,"props":1755,"children":1756},{},[1757],{"type":43,"value":1758},"Raw Foundry\u002FAzure OpenAI model deployment",{"type":38,"tag":1691,"props":1760,"children":1761},{},[1762],{"type":38,"tag":65,"props":1763,"children":1765},{"className":1764},[],[1766],{"type":43,"value":1767},"\"model:\u003Cdeployment-name>\"",{"type":38,"tag":46,"props":1769,"children":1770},{},[1771],{"type":43,"value":1772},"If nothing is found, ask the user once for the agent identifier.",{"type":38,"tag":46,"props":1774,"children":1775},{},[1776,1778,1783,1785,1791,1793,1798,1799,1805,1807,1812],{"type":43,"value":1777},"For Foundry Prompt Agents authored in the Sandbox portal, do not copy\u002Fpaste the\ninstructions into a file manually. After ",{"type":38,"tag":65,"props":1779,"children":1781},{"className":1780},[],[1782],{"type":43,"value":116},{"type":43,"value":1784}," contains ",{"type":38,"tag":65,"props":1786,"children":1788},{"className":1787},[],[1789],{"type":43,"value":1790},"agent: name:version",{"type":43,"value":1792}," and the correct project endpoint is available from\n",{"type":38,"tag":65,"props":1794,"children":1796},{"className":1795},[],[1797],{"type":43,"value":116},{"type":43,"value":173},{"type":38,"tag":65,"props":1800,"children":1802},{"className":1801},[],[1803],{"type":43,"value":1804},"AZURE_AI_FOUNDRY_PROJECT_ENDPOINT",{"type":43,"value":1806},", or the active\n",{"type":38,"tag":65,"props":1808,"children":1810},{"className":1809},[],[1811],{"type":43,"value":155},{"type":43,"value":1813},", run:",{"type":38,"tag":329,"props":1815,"children":1817},{"className":377,"code":1816,"language":379,"meta":337,"style":337},"agentops prompt pull\n",[1818],{"type":38,"tag":65,"props":1819,"children":1820},{"__ignoreMap":337},[1821],{"type":38,"tag":385,"props":1822,"children":1823},{"class":387,"line":388},[1824,1828,1833],{"type":38,"tag":385,"props":1825,"children":1826},{"style":414},[1827],{"type":43,"value":1583},{"type":38,"tag":385,"props":1829,"children":1830},{"style":420},[1831],{"type":43,"value":1832}," prompt",{"type":38,"tag":385,"props":1834,"children":1835},{"style":420},[1836],{"type":43,"value":1837}," pull\n",{"type":38,"tag":46,"props":1839,"children":1840},{},[1841,1843,1849,1851,1857,1859,1864,1866,1872,1874,1880],{"type":43,"value":1842},"This writes ",{"type":38,"tag":65,"props":1844,"children":1846},{"className":1845},[],[1847],{"type":43,"value":1848},".agentops\u002Fprompts\u002F\u003Cagent-name>.prompt.md",{"type":43,"value":1850}," by default, updates\n",{"type":38,"tag":65,"props":1852,"children":1854},{"className":1853},[],[1855],{"type":43,"value":1856},"prompt_file",{"type":43,"value":1858}," in ",{"type":38,"tag":65,"props":1860,"children":1862},{"className":1861},[],[1863],{"type":43,"value":116},{"type":43,"value":1865}," when needed, prints the resolved endpoint and\nagent version before writing, validates that the Foundry definition is a prompt\nagent, and refuses to overwrite changed prompt files unless ",{"type":38,"tag":65,"props":1867,"children":1869},{"className":1868},[],[1870],{"type":43,"value":1871},"--force",{"type":43,"value":1873}," is used.\nUse ",{"type":38,"tag":65,"props":1875,"children":1877},{"className":1876},[],[1878],{"type":43,"value":1879},"--out",{"type":43,"value":1881}," only when the repository already has a stronger prompt-file\nconvention.",{"type":38,"tag":82,"props":1883,"children":1885},{"id":1884},"step-3-make-sure-the-dataset-exists",[1886],{"type":43,"value":1887},"Step 3 - Make sure the dataset exists",{"type":38,"tag":46,"props":1889,"children":1890},{},[1891,1896,1898,1904,1906,1912,1914,1920,1921,1927,1929,1935,1937,1942],{"type":38,"tag":65,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":43,"value":116},{"type":43,"value":1897}," points to a JSONL file (default\n",{"type":38,"tag":65,"props":1899,"children":1901},{"className":1900},[],[1902],{"type":43,"value":1903},".agentops\u002Fdata\u002Fsmoke.jsonl",{"type":43,"value":1905},"). Each row needs at least ",{"type":38,"tag":65,"props":1907,"children":1909},{"className":1908},[],[1910],{"type":43,"value":1911},"input",{"type":43,"value":1913}," and a label\nthat maps to the metric you care about (",{"type":38,"tag":65,"props":1915,"children":1917},{"className":1916},[],[1918],{"type":43,"value":1919},"expected",{"type":43,"value":173},{"type":38,"tag":65,"props":1922,"children":1924},{"className":1923},[],[1925],{"type":43,"value":1926},"context",{"type":43,"value":1928},",\n",{"type":38,"tag":65,"props":1930,"children":1932},{"className":1931},[],[1933],{"type":43,"value":1934},"tool_calls",{"type":43,"value":1936},"...). If the dataset is empty or unrelated, run the\n",{"type":38,"tag":65,"props":1938,"children":1940},{"className":1939},[],[1941],{"type":43,"value":1619},{"type":43,"value":1943}," skill before running the eval.",{"type":38,"tag":82,"props":1945,"children":1947},{"id":1946},"step-4-run-the-evaluation",[1948],{"type":43,"value":1949},"Step 4 - Run the evaluation",{"type":38,"tag":329,"props":1951,"children":1953},{"className":377,"code":1952,"language":379,"meta":337,"style":337},"agentops eval run\n",[1954],{"type":38,"tag":65,"props":1955,"children":1956},{"__ignoreMap":337},[1957],{"type":38,"tag":385,"props":1958,"children":1959},{"class":387,"line":388},[1960,1964,1968],{"type":38,"tag":385,"props":1961,"children":1962},{"style":414},[1963],{"type":43,"value":1583},{"type":38,"tag":385,"props":1965,"children":1966},{"style":420},[1967],{"type":43,"value":1588},{"type":38,"tag":385,"props":1969,"children":1970},{"style":420},[1971],{"type":43,"value":1972}," run\n",{"type":38,"tag":46,"props":1974,"children":1975},{},[1976],{"type":43,"value":1977},"Optional flags:",{"type":38,"tag":1979,"props":1980,"children":1981},"ul",{},[1982,1999],{"type":38,"tag":93,"props":1983,"children":1984},{},[1985,1991,1993,1998],{"type":38,"tag":65,"props":1986,"children":1988},{"className":1987},[],[1989],{"type":43,"value":1990},"--config \u003Cpath>",{"type":43,"value":1992}," - point at a different ",{"type":38,"tag":65,"props":1994,"children":1996},{"className":1995},[],[1997],{"type":43,"value":116},{"type":43,"value":105},{"type":38,"tag":93,"props":2000,"children":2001},{},[2002,2008,2010,2016,2018,2024,2026,2032],{"type":38,"tag":65,"props":2003,"children":2005},{"className":2004},[],[2006],{"type":43,"value":2007},"--output \u003Cdir>",{"type":43,"value":2009}," - choose where to write ",{"type":38,"tag":65,"props":2011,"children":2013},{"className":2012},[],[2014],{"type":43,"value":2015},"results.json",{"type":43,"value":2017}," and ",{"type":38,"tag":65,"props":2019,"children":2021},{"className":2020},[],[2022],{"type":43,"value":2023},"report.md",{"type":43,"value":2025},"\n(defaults to ",{"type":38,"tag":65,"props":2027,"children":2029},{"className":2028},[],[2030],{"type":43,"value":2031},".agentops\u002Fresults\u002F\u003Ctimestamp>\u002F",{"type":43,"value":267},{"type":38,"tag":46,"props":2034,"children":2035},{},[2036],{"type":43,"value":2037},"Exit codes:",{"type":38,"tag":1979,"props":2039,"children":2040},{},[2041,2052,2063],{"type":38,"tag":93,"props":2042,"children":2043},{},[2044,2050],{"type":38,"tag":65,"props":2045,"children":2047},{"className":2046},[],[2048],{"type":43,"value":2049},"0",{"type":43,"value":2051}," - succeeded and all thresholds passed",{"type":38,"tag":93,"props":2053,"children":2054},{},[2055,2061],{"type":38,"tag":65,"props":2056,"children":2058},{"className":2057},[],[2059],{"type":43,"value":2060},"2",{"type":43,"value":2062}," - succeeded but at least one threshold failed (gate-friendly)",{"type":38,"tag":93,"props":2064,"children":2065},{},[2066,2072],{"type":38,"tag":65,"props":2067,"children":2069},{"className":2068},[],[2070],{"type":43,"value":2071},"1",{"type":43,"value":2073}," - runtime\u002Fconfiguration error",{"type":38,"tag":82,"props":2075,"children":2077},{"id":2076},"step-4b-pick-the-right-execution-path",[2078],{"type":43,"value":2079},"Step 4b - Pick the right execution path",{"type":38,"tag":1656,"props":2081,"children":2082},{},[2083,2109],{"type":38,"tag":1660,"props":2084,"children":2085},{},[2086],{"type":38,"tag":1664,"props":2087,"children":2088},{},[2089,2094,2099,2104],{"type":38,"tag":1668,"props":2090,"children":2091},{},[2092],{"type":43,"value":2093},"Target",{"type":38,"tag":1668,"props":2095,"children":2096},{},[2097],{"type":43,"value":2098},"Foundry server-side eval through AgentOps",{"type":38,"tag":1668,"props":2100,"children":2101},{},[2102],{"type":43,"value":2103},"AgentOps local runner",{"type":38,"tag":1668,"props":2105,"children":2106},{},[2107],{"type":43,"value":2108},"Default guidance",{"type":38,"tag":1684,"props":2110,"children":2111},{},[2112,2146,2176,2197,2217],{"type":38,"tag":1664,"props":2113,"children":2114},{},[2115,2127,2136,2141],{"type":38,"tag":1691,"props":2116,"children":2117},{},[2118,2120,2125],{"type":43,"value":2119},"Foundry Prompt Agent (",{"type":38,"tag":65,"props":2121,"children":2123},{"className":2122},[],[2124],{"type":43,"value":78},{"type":43,"value":2126},")",{"type":38,"tag":1691,"props":2128,"children":2129},{},[2130],{"type":38,"tag":65,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":43,"value":2135},"execution: cloud",{"type":38,"tag":1691,"props":2137,"children":2138},{},[2139],{"type":43,"value":2140},"yes",{"type":38,"tag":1691,"props":2142,"children":2143},{},[2144],{"type":43,"value":2145},"Use cloud when the user wants the official Foundry-hosted eval record; use local for fast feedback or fallback.",{"type":38,"tag":1664,"props":2147,"children":2148},{},[2149,2154,2159,2163],{"type":38,"tag":1691,"props":2150,"children":2151},{},[2152],{"type":43,"value":2153},"Foundry Hosted Agent URL",{"type":38,"tag":1691,"props":2155,"children":2156},{},[2157],{"type":43,"value":2158},"no",{"type":38,"tag":1691,"props":2160,"children":2161},{},[2162],{"type":43,"value":2140},{"type":38,"tag":1691,"props":2164,"children":2165},{},[2166,2168,2174],{"type":43,"value":2167},"Use local runner; optionally set ",{"type":38,"tag":65,"props":2169,"children":2171},{"className":2170},[],[2172],{"type":43,"value":2173},"publish: true",{"type":43,"value":2175}," to upload local metrics to Classic Foundry.",{"type":38,"tag":1664,"props":2177,"children":2178},{},[2179,2184,2188,2192],{"type":38,"tag":1691,"props":2180,"children":2181},{},[2182],{"type":43,"value":2183},"Generic HTTP\u002FJSON endpoint",{"type":38,"tag":1691,"props":2185,"children":2186},{},[2187],{"type":43,"value":2158},{"type":38,"tag":1691,"props":2189,"children":2190},{},[2191],{"type":43,"value":2140},{"type":38,"tag":1691,"props":2193,"children":2194},{},[2195],{"type":43,"value":2196},"Use local runner.",{"type":38,"tag":1664,"props":2198,"children":2199},{},[2200,2205,2209,2213],{"type":38,"tag":1691,"props":2201,"children":2202},{},[2203],{"type":43,"value":2204},"Raw model deployment",{"type":38,"tag":1691,"props":2206,"children":2207},{},[2208],{"type":43,"value":2158},{"type":38,"tag":1691,"props":2210,"children":2211},{},[2212],{"type":43,"value":2140},{"type":38,"tag":1691,"props":2214,"children":2215},{},[2216],{"type":43,"value":2196},{"type":38,"tag":1664,"props":2218,"children":2219},{},[2220,2233,2242,2247],{"type":38,"tag":1691,"props":2221,"children":2222},{},[2223,2225,2231],{"type":43,"value":2224},"azd ",{"type":38,"tag":65,"props":2226,"children":2228},{"className":2227},[],[2229],{"type":43,"value":2230},"eval.yaml",{"type":43,"value":2232}," recipe",{"type":38,"tag":1691,"props":2234,"children":2235},{},[2236],{"type":38,"tag":65,"props":2237,"children":2239},{"className":2238},[],[2240],{"type":43,"value":2241},"execution: azd",{"type":38,"tag":1691,"props":2243,"children":2244},{},[2245],{"type":43,"value":2246},"no fallback",{"type":38,"tag":1691,"props":2248,"children":2249},{},[2250,2252,2258,2260,2266],{"type":43,"value":2251},"Use ",{"type":38,"tag":65,"props":2253,"children":2255},{"className":2254},[],[2256],{"type":43,"value":2257},"agentops eval init",{"type":43,"value":2259}," \u002F ",{"type":38,"tag":65,"props":2261,"children":2263},{"className":2262},[],[2264],{"type":43,"value":2265},"azd ai agent eval",{"type":43,"value":2267}," for Foundry prompt\u002Fhosted agents when the user wants the public-preview azd AI agent eval workflow.",{"type":38,"tag":46,"props":2269,"children":2270},{},[2271,2273,2278,2279,2284],{"type":43,"value":2272},"For prompt-agent CI gates, prefer AgentOps cloud eval because Foundry executes\nthe managed eval while AgentOps enforces thresholds and writes normalized\n",{"type":38,"tag":65,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":43,"value":2015},{"type":43,"value":2259},{"type":38,"tag":65,"props":2280,"children":2282},{"className":2281},[],[2283],{"type":43,"value":2023},{"type":43,"value":2285}," artifacts. The official AI Agent Evaluation GitHub\nAction or Azure DevOps extension is still useful for standalone platform-native\nvalidation, but do not substitute it for the AgentOps PR gate when the user needs\nthreshold enforcement, baselines, Doctor readiness, release evidence, or local\nfallback.",{"type":38,"tag":46,"props":2287,"children":2288},{},[2289,2291,2296,2298,2303,2305,2311],{"type":43,"value":2290},"When an ",{"type":38,"tag":65,"props":2292,"children":2294},{"className":2293},[],[2295],{"type":43,"value":2230},{"type":43,"value":2297}," includes Rubric evaluator dimensions, keep thresholds in\n",{"type":38,"tag":65,"props":2299,"children":2301},{"className":2300},[],[2302],{"type":43,"value":116},{"type":43,"value":2304}," aligned to the dimension metric names (for example\n",{"type":38,"tag":65,"props":2306,"children":2308},{"className":2307},[],[2309],{"type":43,"value":2310},"booking_accuracy: \">=0.8\"",{"type":43,"value":2312},"). AgentOps binds Rubric\u002Fcustom dimensions literally\nand fails closed when a configured threshold has no matching emitted metric.",{"type":38,"tag":82,"props":2314,"children":2316},{"id":2315},"step-5-inspect-results-and-release-evidence",[2317],{"type":43,"value":2318},"Step 5 - Inspect results and release evidence",{"type":38,"tag":329,"props":2320,"children":2322},{"className":377,"code":2321,"language":379,"meta":337,"style":337},"agentops report generate                   # regenerate report.md from latest results.json\nagentops report generate --in \u003Cresults.json>\n",[2323],{"type":38,"tag":65,"props":2324,"children":2325},{"__ignoreMap":337},[2326,2348],{"type":38,"tag":385,"props":2327,"children":2328},{"class":387,"line":388},[2329,2333,2338,2343],{"type":38,"tag":385,"props":2330,"children":2331},{"style":414},[2332],{"type":43,"value":1583},{"type":38,"tag":385,"props":2334,"children":2335},{"style":420},[2336],{"type":43,"value":2337}," report",{"type":38,"tag":385,"props":2339,"children":2340},{"style":420},[2341],{"type":43,"value":2342}," generate",{"type":38,"tag":385,"props":2344,"children":2345},{"style":392},[2346],{"type":43,"value":2347},"                   # regenerate report.md from latest results.json\n",{"type":38,"tag":385,"props":2349,"children":2350},{"class":387,"line":398},[2351,2355,2359,2363,2368,2373,2378,2383],{"type":38,"tag":385,"props":2352,"children":2353},{"style":414},[2354],{"type":43,"value":1583},{"type":38,"tag":385,"props":2356,"children":2357},{"style":420},[2358],{"type":43,"value":2337},{"type":38,"tag":385,"props":2360,"children":2361},{"style":420},[2362],{"type":43,"value":2342},{"type":38,"tag":385,"props":2364,"children":2365},{"style":420},[2366],{"type":43,"value":2367}," --in",{"type":38,"tag":385,"props":2369,"children":2370},{"style":408},[2371],{"type":43,"value":2372}," \u003C",{"type":38,"tag":385,"props":2374,"children":2375},{"style":420},[2376],{"type":43,"value":2377},"results.jso",{"type":38,"tag":385,"props":2379,"children":2380},{"style":402},[2381],{"type":43,"value":2382},"n",{"type":38,"tag":385,"props":2384,"children":2385},{"style":408},[2386],{"type":43,"value":2387},">\n",{"type":38,"tag":46,"props":2389,"children":2390},{},[2391,2393,2399,2401,2406,2408,2414,2416,2421],{"type":43,"value":2392},"Open ",{"type":38,"tag":65,"props":2394,"children":2396},{"className":2395},[],[2397],{"type":43,"value":2398},".agentops\u002Fresults\u002Flatest\u002Freport.md",{"type":43,"value":2400},". To compare two runs, hand both\n",{"type":38,"tag":65,"props":2402,"children":2404},{"className":2403},[],[2405],{"type":43,"value":2015},{"type":43,"value":2407}," files to the user or run the next eval with\n",{"type":38,"tag":65,"props":2409,"children":2411},{"className":2410},[],[2412],{"type":43,"value":2413},"--baseline \u003Cprevious-results.json>",{"type":43,"value":2415}," so AgentOps adds a ",{"type":38,"tag":57,"props":2417,"children":2418},{},[2419],{"type":43,"value":2420},"Comparison vs\nBaseline",{"type":43,"value":2422}," section to the report.",{"type":38,"tag":46,"props":2424,"children":2425},{},[2426],{"type":43,"value":2427},"For production promotion, generate the Doctor evidence pack:",{"type":38,"tag":329,"props":2429,"children":2431},{"className":377,"code":2430,"language":379,"meta":337,"style":337},"agentops doctor --evidence-pack\n",[2432],{"type":38,"tag":65,"props":2433,"children":2434},{"__ignoreMap":337},[2435],{"type":38,"tag":385,"props":2436,"children":2437},{"class":387,"line":388},[2438,2442,2447],{"type":38,"tag":385,"props":2439,"children":2440},{"style":414},[2441],{"type":43,"value":1583},{"type":38,"tag":385,"props":2443,"children":2444},{"style":420},[2445],{"type":43,"value":2446}," doctor",{"type":38,"tag":385,"props":2448,"children":2449},{"style":420},[2450],{"type":43,"value":2451}," --evidence-pack\n",{"type":38,"tag":46,"props":2453,"children":2454},{},[2455,2456,2462,2464,2470,2472,2477],{"type":43,"value":2392},{"type":38,"tag":65,"props":2457,"children":2459},{"className":2458},[],[2460],{"type":43,"value":2461},".agentops\u002Frelease\u002Flatest\u002Fevidence.md",{"type":43,"value":2463},". It summarizes eval, baseline,\nDoctor, workflow, Foundry, monitoring, AI Landing Zone, and trace-regression\nreadiness without creating a second exit-code contract. If the repo has ASSERT,\nACS, or red-team evidence artifacts, use the ",{"type":38,"tag":65,"props":2465,"children":2467},{"className":2466},[],[2468],{"type":43,"value":2469},"agentops-governance",{"type":43,"value":2471}," skill to wire\ntheir paths into ",{"type":38,"tag":65,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":43,"value":116},{"type":43,"value":2478}," before generating the evidence pack.",{"type":38,"tag":82,"props":2480,"children":2482},{"id":2481},"step-5b-optional-promote-reviewed-traces-to-regression-rows",[2483],{"type":43,"value":2484},"Step 5b - (Optional) Promote reviewed traces to regression rows",{"type":38,"tag":46,"props":2486,"children":2487},{},[2488],{"type":43,"value":2489},"If the user has exported Foundry\u002FApp Insights traces, preview candidate\nregression rows first:",{"type":38,"tag":329,"props":2491,"children":2493},{"className":377,"code":2492,"language":379,"meta":337,"style":337},"agentops eval promote-traces --source \u003Ctraces.jsonl>\n",[2494],{"type":38,"tag":65,"props":2495,"children":2496},{"__ignoreMap":337},[2497],{"type":38,"tag":385,"props":2498,"children":2499},{"class":387,"line":388},[2500,2504,2508,2513,2518,2522,2527,2532],{"type":38,"tag":385,"props":2501,"children":2502},{"style":414},[2503],{"type":43,"value":1583},{"type":38,"tag":385,"props":2505,"children":2506},{"style":420},[2507],{"type":43,"value":1588},{"type":38,"tag":385,"props":2509,"children":2510},{"style":420},[2511],{"type":43,"value":2512}," promote-traces",{"type":38,"tag":385,"props":2514,"children":2515},{"style":420},[2516],{"type":43,"value":2517}," --source",{"type":38,"tag":385,"props":2519,"children":2520},{"style":408},[2521],{"type":43,"value":2372},{"type":38,"tag":385,"props":2523,"children":2524},{"style":420},[2525],{"type":43,"value":2526},"traces.json",{"type":38,"tag":385,"props":2528,"children":2529},{"style":402},[2530],{"type":43,"value":2531},"l",{"type":38,"tag":385,"props":2533,"children":2534},{"style":408},[2535],{"type":43,"value":2387},{"type":38,"tag":46,"props":2537,"children":2538},{},[2539],{"type":43,"value":2540},"Only write files after review:",{"type":38,"tag":329,"props":2542,"children":2544},{"className":377,"code":2543,"language":379,"meta":337,"style":337},"agentops eval promote-traces --source \u003Ctraces.jsonl> --apply\n",[2545],{"type":38,"tag":65,"props":2546,"children":2547},{"__ignoreMap":337},[2548],{"type":38,"tag":385,"props":2549,"children":2550},{"class":387,"line":388},[2551,2555,2559,2563,2567,2571,2575,2579,2584],{"type":38,"tag":385,"props":2552,"children":2553},{"style":414},[2554],{"type":43,"value":1583},{"type":38,"tag":385,"props":2556,"children":2557},{"style":420},[2558],{"type":43,"value":1588},{"type":38,"tag":385,"props":2560,"children":2561},{"style":420},[2562],{"type":43,"value":2512},{"type":38,"tag":385,"props":2564,"children":2565},{"style":420},[2566],{"type":43,"value":2517},{"type":38,"tag":385,"props":2568,"children":2569},{"style":408},[2570],{"type":43,"value":2372},{"type":38,"tag":385,"props":2572,"children":2573},{"style":420},[2574],{"type":43,"value":2526},{"type":38,"tag":385,"props":2576,"children":2577},{"style":402},[2578],{"type":43,"value":2531},{"type":38,"tag":385,"props":2580,"children":2581},{"style":408},[2582],{"type":43,"value":2583},">",{"type":38,"tag":385,"props":2585,"children":2586},{"style":420},[2587],{"type":43,"value":2588}," --apply\n",{"type":38,"tag":46,"props":2590,"children":2591},{},[2592,2594,2600,2602,2608],{"type":43,"value":2593},"Default ",{"type":38,"tag":65,"props":2595,"children":2597},{"className":2596},[],[2598],{"type":43,"value":2599},"self-similarity",{"type":43,"value":2601}," labels are for drift detection, not human-verified\nground truth. Use ",{"type":38,"tag":65,"props":2603,"children":2605},{"className":2604},[],[2606],{"type":43,"value":2607},"--label-mode pending",{"type":43,"value":2609}," when reviewers must fill expected\nanswers before the dataset gates releases.",{"type":38,"tag":82,"props":2611,"children":2613},{"id":2612},"step-6-optional-foundry-execution-visibility",[2614],{"type":43,"value":2615},"Step 6 - (Optional) Foundry execution \u002F visibility",{"type":38,"tag":46,"props":2617,"children":2618},{},[2619,2621,2627,2629,2634,2636,2642],{"type":43,"value":2620},"Two modes are supported. Both write a deep-link into\n",{"type":38,"tag":65,"props":2622,"children":2624},{"className":2623},[],[2625],{"type":43,"value":2626},".agentops\u002Fresults\u002Flatest\u002Fcloud_evaluation.json",{"type":43,"value":2628}," and require\n",{"type":38,"tag":65,"props":2630,"children":2632},{"className":2631},[],[2633],{"type":43,"value":1804},{"type":43,"value":2635}," (or the inline ",{"type":38,"tag":65,"props":2637,"children":2639},{"className":2638},[],[2640],{"type":43,"value":2641},"project_endpoint",{"type":43,"value":267},{"type":38,"tag":46,"props":2644,"children":2645},{},[2646,2651],{"type":38,"tag":57,"props":2647,"children":2648},{},[2649],{"type":43,"value":2650},"Classic Foundry Evaluations panel",{"type":43,"value":2652}," (works for any target kind):\nAgentOps runs locally first, then uploads the metrics it computed.",{"type":38,"tag":329,"props":2654,"children":2658},{"className":2655,"code":2656,"language":2657,"meta":337,"style":337},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","execution: local\npublish: true\n# project_endpoint: \"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\"\n","yaml",[2659],{"type":38,"tag":65,"props":2660,"children":2661},{"__ignoreMap":337},[2662,2679,2697],{"type":38,"tag":385,"props":2663,"children":2664},{"class":387,"line":388},[2665,2670,2674],{"type":38,"tag":385,"props":2666,"children":2668},{"style":2667},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[2669],{"type":43,"value":1548},{"type":38,"tag":385,"props":2671,"children":2672},{"style":408},[2673],{"type":43,"value":1654},{"type":38,"tag":385,"props":2675,"children":2676},{"style":420},[2677],{"type":43,"value":2678}," local\n",{"type":38,"tag":385,"props":2680,"children":2681},{"class":387,"line":398},[2682,2687,2691],{"type":38,"tag":385,"props":2683,"children":2684},{"style":2667},[2685],{"type":43,"value":2686},"publish",{"type":38,"tag":385,"props":2688,"children":2689},{"style":408},[2690],{"type":43,"value":1654},{"type":38,"tag":385,"props":2692,"children":2694},{"style":2693},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[2695],{"type":43,"value":2696}," true\n",{"type":38,"tag":385,"props":2698,"children":2699},{"class":387,"line":537},[2700],{"type":38,"tag":385,"props":2701,"children":2702},{"style":392},[2703],{"type":43,"value":2704},"# project_endpoint: \"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\"\n",{"type":38,"tag":46,"props":2706,"children":2707},{},[2708,2713,2715,2720,2722,2727],{"type":38,"tag":57,"props":2709,"children":2710},{},[2711],{"type":43,"value":2712},"New Foundry Evaluations panel",{"type":43,"value":2714}," (preview): Foundry runs the agent +\nevaluators server-side via the OpenAI Evals API. Only works for\n",{"type":38,"tag":65,"props":2716,"children":2718},{"className":2717},[],[2719],{"type":43,"value":78},{"type":43,"value":2721}," Foundry agents. ",{"type":38,"tag":65,"props":2723,"children":2725},{"className":2724},[],[2726],{"type":43,"value":2686},{"type":43,"value":2728}," is implicit - a cloud run is\nalways recorded by Foundry. The local JSONL remains the dataset source of\ntruth; AgentOps syncs it to Foundry Data\u002FDatasets by default and uses that\ndataset version in the Evals run.",{"type":38,"tag":329,"props":2730,"children":2732},{"className":2655,"code":2731,"language":2657,"meta":337,"style":337},"execution: cloud\n# project_endpoint: \"https:\u002F\u002F\u003Cresource>.services.ai.azure.com\u002Fapi\u002Fprojects\u002F\u003Cp>\"\ndataset_sync:\n  mode: auto            # sync local JSONL to Foundry Data\u002FDatasets\n",[2733],{"type":38,"tag":65,"props":2734,"children":2735},{"__ignoreMap":337},[2736,2752,2759,2772],{"type":38,"tag":385,"props":2737,"children":2738},{"class":387,"line":388},[2739,2743,2747],{"type":38,"tag":385,"props":2740,"children":2741},{"style":2667},[2742],{"type":43,"value":1548},{"type":38,"tag":385,"props":2744,"children":2745},{"style":408},[2746],{"type":43,"value":1654},{"type":38,"tag":385,"props":2748,"children":2749},{"style":420},[2750],{"type":43,"value":2751}," cloud\n",{"type":38,"tag":385,"props":2753,"children":2754},{"class":387,"line":398},[2755],{"type":38,"tag":385,"props":2756,"children":2757},{"style":392},[2758],{"type":43,"value":2704},{"type":38,"tag":385,"props":2760,"children":2761},{"class":387,"line":537},[2762,2767],{"type":38,"tag":385,"props":2763,"children":2764},{"style":2667},[2765],{"type":43,"value":2766},"dataset_sync",{"type":38,"tag":385,"props":2768,"children":2769},{"style":408},[2770],{"type":43,"value":2771},":\n",{"type":38,"tag":385,"props":2773,"children":2774},{"class":387,"line":616},[2775,2780,2784,2789],{"type":38,"tag":385,"props":2776,"children":2777},{"style":2667},[2778],{"type":43,"value":2779},"  mode",{"type":38,"tag":385,"props":2781,"children":2782},{"style":408},[2783],{"type":43,"value":1654},{"type":38,"tag":385,"props":2785,"children":2786},{"style":420},[2787],{"type":43,"value":2788}," auto",{"type":38,"tag":385,"props":2790,"children":2791},{"style":392},[2792],{"type":43,"value":2793},"            # sync local JSONL to Foundry Data\u002FDatasets\n",{"type":38,"tag":46,"props":2795,"children":2796},{},[2797,2799,2805,2807,2812],{"type":43,"value":2798},"With ",{"type":38,"tag":65,"props":2800,"children":2802},{"className":2801},[],[2803],{"type":43,"value":2804},"execution: local",{"type":43,"value":2806}," and no ",{"type":38,"tag":65,"props":2808,"children":2810},{"className":2809},[],[2811],{"type":43,"value":2173},{"type":43,"value":2813},", AgentOps runs locally\nand only writes local artifacts.",{"type":38,"tag":46,"props":2815,"children":2816},{},[2817,2819,2824,2826,2832],{"type":43,"value":2818},"After a cloud run, inspect ",{"type":38,"tag":65,"props":2820,"children":2822},{"className":2821},[],[2823],{"type":43,"value":2626},{"type":43,"value":2825},".\nIts ",{"type":38,"tag":65,"props":2827,"children":2829},{"className":2828},[],[2830],{"type":43,"value":2831},"dataset",{"type":43,"value":2833}," block explains whether the run used inline rows or a Foundry\ndataset reference.",{"type":38,"tag":82,"props":2835,"children":2837},{"id":2836},"tips",[2838],{"type":43,"value":2839},"Tips",{"type":38,"tag":1979,"props":2841,"children":2842},{},[2843,2856,2938,2973],{"type":38,"tag":93,"props":2844,"children":2845},{},[2846,2848,2854],{"type":43,"value":2847},"Evaluators are auto-selected from the agent type and dataset columns.\nOverride only when needed via the ",{"type":38,"tag":65,"props":2849,"children":2851},{"className":2850},[],[2852],{"type":43,"value":2853},"evaluators:",{"type":43,"value":2855}," block - most users do\nnot need it.",{"type":38,"tag":93,"props":2857,"children":2858},{},[2859,2861,2867,2869],{"type":43,"value":2860},"Set thresholds in ",{"type":38,"tag":65,"props":2862,"children":2864},{"className":2863},[],[2865],{"type":43,"value":2866},"thresholds:",{"type":43,"value":2868}," to gate CI:\n",{"type":38,"tag":329,"props":2870,"children":2872},{"className":2655,"code":2871,"language":2657,"meta":337,"style":337},"thresholds:\n  coherence: \">=3\"\n  avg_latency_seconds: \"\u003C=10\"\n",[2873],{"type":38,"tag":65,"props":2874,"children":2875},{"__ignoreMap":337},[2876,2888,2913],{"type":38,"tag":385,"props":2877,"children":2878},{"class":387,"line":388},[2879,2884],{"type":38,"tag":385,"props":2880,"children":2881},{"style":2667},[2882],{"type":43,"value":2883},"thresholds",{"type":38,"tag":385,"props":2885,"children":2886},{"style":408},[2887],{"type":43,"value":2771},{"type":38,"tag":385,"props":2889,"children":2890},{"class":387,"line":398},[2891,2896,2900,2904,2909],{"type":38,"tag":385,"props":2892,"children":2893},{"style":2667},[2894],{"type":43,"value":2895},"  coherence",{"type":38,"tag":385,"props":2897,"children":2898},{"style":408},[2899],{"type":43,"value":1654},{"type":38,"tag":385,"props":2901,"children":2902},{"style":408},[2903],{"type":43,"value":558},{"type":38,"tag":385,"props":2905,"children":2906},{"style":420},[2907],{"type":43,"value":2908},">=3",{"type":38,"tag":385,"props":2910,"children":2911},{"style":408},[2912],{"type":43,"value":1076},{"type":38,"tag":385,"props":2914,"children":2915},{"class":387,"line":537},[2916,2921,2925,2929,2934],{"type":38,"tag":385,"props":2917,"children":2918},{"style":2667},[2919],{"type":43,"value":2920},"  avg_latency_seconds",{"type":38,"tag":385,"props":2922,"children":2923},{"style":408},[2924],{"type":43,"value":1654},{"type":38,"tag":385,"props":2926,"children":2927},{"style":408},[2928],{"type":43,"value":558},{"type":38,"tag":385,"props":2930,"children":2931},{"style":420},[2932],{"type":43,"value":2933},"\u003C=10",{"type":38,"tag":385,"props":2935,"children":2936},{"style":408},[2937],{"type":43,"value":1076},{"type":38,"tag":93,"props":2939,"children":2940},{},[2941,2943,2949,2951,2957,2959,2965,2966,2972],{"type":43,"value":2942},"For HTTP\u002FJSON agents that need auth, set\n",{"type":38,"tag":65,"props":2944,"children":2946},{"className":2945},[],[2947],{"type":43,"value":2948},"auth_header_env: MY_TOKEN_VAR",{"type":43,"value":2950}," and AgentOps adds\n",{"type":38,"tag":65,"props":2952,"children":2954},{"className":2953},[],[2955],{"type":43,"value":2956},"Authorization: Bearer $MY_TOKEN_VAR",{"type":43,"value":2958},". For a shared-secret gate, override the\nheader with ",{"type":38,"tag":65,"props":2960,"children":2962},{"className":2961},[],[2963],{"type":43,"value":2964},"auth_header_name: X-API-KEY",{"type":43,"value":2017},{"type":38,"tag":65,"props":2967,"children":2969},{"className":2968},[],[2970],{"type":43,"value":2971},"auth_value_template: \"{token}\"",{"type":43,"value":105},{"type":38,"tag":93,"props":2974,"children":2975},{},[2976,2978,2984,2986,2992,2994,3000,3002,3008,3010,3016,3018,3024,3026,3031,3033,3039,3041,3047,3049,3055,3057,3063,3065,3071],{"type":43,"value":2977},"For streaming HTTP agents (e.g. an SSE ",{"type":38,"tag":65,"props":2979,"children":2981},{"className":2980},[],[2982],{"type":43,"value":2983},"text\u002Fevent-stream",{"type":43,"value":2985}," endpoint), set\n",{"type":38,"tag":65,"props":2987,"children":2989},{"className":2988},[],[2990],{"type":43,"value":2991},"response_mode: sse",{"type":43,"value":2993}," (each ",{"type":38,"tag":65,"props":2995,"children":2997},{"className":2996},[],[2998],{"type":43,"value":2999},"data:",{"type":43,"value":3001}," line) or ",{"type":38,"tag":65,"props":3003,"children":3005},{"className":3004},[],[3006],{"type":43,"value":3007},"response_mode: text",{"type":43,"value":3009}," (raw\nstreamed text). Use the optional ",{"type":38,"tag":65,"props":3011,"children":3013},{"className":3012},[],[3014],{"type":43,"value":3015},"stream:",{"type":43,"value":3017}," block to tune aggregation:\n",{"type":38,"tag":65,"props":3019,"children":3021},{"className":3020},[],[3022],{"type":43,"value":3023},"text_field",{"type":43,"value":3025}," (dot-path to the token text when ",{"type":38,"tag":65,"props":3027,"children":3029},{"className":3028},[],[3030],{"type":43,"value":2999},{"type":43,"value":3032}," lines are JSON),\n",{"type":38,"tag":65,"props":3034,"children":3036},{"className":3035},[],[3037],{"type":43,"value":3038},"done_marker",{"type":43,"value":3040}," (e.g. ",{"type":38,"tag":65,"props":3042,"children":3044},{"className":3043},[],[3045],{"type":43,"value":3046},"[DONE]",{"type":43,"value":3048},"), and ",{"type":38,"tag":65,"props":3050,"children":3052},{"className":3051},[],[3053],{"type":43,"value":3054},"strip_leading_token: true",{"type":43,"value":3056}," (drop a leading\n",{"type":38,"tag":65,"props":3058,"children":3060},{"className":3059},[],[3061],{"type":43,"value":3062},"conversation_id",{"type":43,"value":3064}," prefix). ",{"type":38,"tag":65,"props":3066,"children":3068},{"className":3067},[],[3069],{"type":43,"value":3070},"response_mode: json",{"type":43,"value":3072}," (default) is unchanged.",{"type":38,"tag":3074,"props":3075,"children":3076},"style",{},[3077],{"type":43,"value":3078},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3080,"total":3259},[3081,3100,3119,3138,3153,3168,3181,3196,3207,3221,3234,3247],{"slug":3082,"name":3082,"fn":3083,"description":3084,"org":3085,"tags":3086,"stars":3097,"repoUrl":3098,"updatedAt":3099},"azure-arg-external-evaluation-policy-author","author and test Azure Resource Graph policies","Use when the user wants to author, design, or test an Azure Policy that queries Azure Resource Graph (ARG) at request-time — i.e. a policy whose deny\u002Faudit decision depends on data from elsewhere in the subscription (sibling\u002Fparent resource state, RG-wide invariants, multi-hop relationships, etc.). Formally called Azure Policy External Evaluation; sometimes referred to colloquially as \"Invoke\". Drives an iterative KQL co-design loop against the user's real subscription via `az graph query`, then emits a policy definition, assignment, `.http` test flow, and an `EXPLANATION.md` companion. Read-only; never provisions anything.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3087,3088,3091,3094],{"name":11,"slug":8,"type":14},{"name":3089,"slug":3090,"type":14},"Compliance","compliance",{"name":3092,"slug":3093,"type":14},"Governance","governance",{"name":3095,"slug":3096,"type":14},"Policy","policy",1686,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-policy","2026-07-12T08:17:48.378432",{"slug":3101,"name":3101,"fn":3102,"description":3103,"org":3104,"tags":3105,"stars":3116,"repoUrl":3117,"updatedAt":3118},"azure-blueprints-migration","migrate Azure Blueprints to Template Specs","Use when a user needs to migrate off Azure Blueprints (definitions and\u002For assignments) to Template Specs and Deployment Stacks before the January 31, 2027 retirement. Covers inventory, export, conversion to Bicep, policy decoupling, Template Spec publishing, Deployment Stack deployment with deny-settings, validation, and cutover.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3106,3107,3110,3113],{"name":11,"slug":8,"type":14},{"name":3108,"slug":3109,"type":14},"Deployment","deployment",{"name":3111,"slug":3112,"type":14},"Infrastructure as Code","infrastructure-as-code",{"name":3114,"slug":3115,"type":14},"Migration","migration",260,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-blueprints","2026-07-12T08:17:49.646405",{"slug":3120,"name":3120,"fn":3121,"description":3122,"org":3123,"tags":3124,"stars":3135,"repoUrl":3136,"updatedAt":3137},"apiview-feedback-resolution","resolve APIView feedback on Azure SDKs","Analyze and resolve APIView review feedback on Azure SDK PRs. **UTILITY SKILL**. USE FOR: APIView comments, API review feedback, SDK API surface changes. DO NOT USE FOR: general code review, non-APIView feedback. INVOKES: azure-sdk-mcp:azsdk_apiview_get_comments, azure-sdk-mcp:azsdk_typespec_customized_code_update.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3125,3128,3129,3132],{"name":3126,"slug":3127,"type":14},"API Development","api-development",{"name":11,"slug":8,"type":14},{"name":3130,"slug":3131,"type":14},"Code Review","code-review",{"name":3133,"slug":3134,"type":14},"Documentation","documentation",133,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-tools","2026-07-12T08:17:43.350876",{"slug":3139,"name":3139,"fn":3140,"description":3141,"org":3142,"tags":3143,"stars":3135,"repoUrl":3136,"updatedAt":3152},"azsdk-common-live-and-recorded-tests","deploy resources and run Azure SDK tests","Deploy test resources and run Azure SDK tests in live, record, or playback mode. WHEN: \"run live tests\", \"run recorded tests\", \"deploy test resources\", \"record tests\", \"run tests in record mode\", \"clean up test resources\", \"run tests against live resources\". DO NOT USE FOR: writing new tests, authoring Bicep templates, playback-only test runs without resource deployment. INVOKES: azure-sdk-mcp:azsdk_package_run_tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3144,3145,3146,3149],{"name":11,"slug":8,"type":14},{"name":3108,"slug":3109,"type":14},{"name":3147,"slug":3148,"type":14},"SDK","sdk",{"name":3150,"slug":3151,"type":14},"Testing","testing","2026-07-12T08:17:44.718943",{"slug":3154,"name":3154,"fn":3155,"description":3156,"org":3157,"tags":3158,"stars":3135,"repoUrl":3136,"updatedAt":3167},"azsdk-common-prepare-release-plan","manage Azure SDK release plan work items","Create, get, update, abandon, and link SDK PRs to release plan work items for Azure SDK releases. **UTILITY SKILL**. USE FOR: \"create release plan\", \"get release plan\", \"update release plan\", \"update API spec in release plan\", \"update SDK details in release plan\", \"abandon release plan\", \"link SDK PR to plan\", \"namespace approval\", \"check release plan status\". DO NOT USE FOR: SDK code generation, pipeline troubleshooting, API review feedback. INVOKES: azure-sdk-mcp:azsdk_create_release_plan, azure-sdk-mcp:azsdk_get_release_plan, azure-sdk-mcp:azsdk_get_release_plan_for_spec_pr, azure-sdk-mcp:azsdk_update_release_plan, azure-sdk-mcp:azsdk_update_api_spec_pull_request_in_release_plan, azure-sdk-mcp:azsdk_update_sdk_details_in_release_plan, azure-sdk-mcp:azsdk_abandon_release_plan, azure-sdk-mcp:azsdk_link_sdk_pull_request_to_release_plan, azure-sdk-mcp:azsdk_link_namespace_approval_issue.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3159,3160,3163,3166],{"name":11,"slug":8,"type":14},{"name":3161,"slug":3162,"type":14},"GitHub","github",{"name":3164,"slug":3165,"type":14},"Project Management","project-management",{"name":3147,"slug":3148,"type":14},"2026-07-12T08:17:38.345387",{"slug":3169,"name":3169,"fn":3170,"description":3171,"org":3172,"tags":3173,"stars":3135,"repoUrl":3136,"updatedAt":3180},"azsdk-common-sdk-release","release Azure SDK packages","Check release readiness and trigger the release pipeline for Azure SDK packages. **UTILITY SKILL**. USE FOR: \"release SDK\", \"trigger release\", \"check release readiness\", \"release pipeline\", \"publish package\", \"ship SDK\". DO NOT USE FOR: SDK development, code generation, pipeline debugging, release plan creation. INVOKES: azure-sdk-mcp:azsdk_release_sdk.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3174,3175,3178,3179],{"name":11,"slug":8,"type":14},{"name":3176,"slug":3177,"type":14},"CI\u002FCD","ci-cd",{"name":3108,"slug":3109,"type":14},{"name":3147,"slug":3148,"type":14},"2026-07-12T08:17:34.27607",{"slug":3182,"name":3182,"fn":3183,"description":3184,"org":3185,"tags":3186,"stars":3135,"repoUrl":3136,"updatedAt":3195},"azure-typespec-author","author and modify Azure TypeSpec API specifications","Authors and modifies Azure TypeSpec (.tsp) API specifications. USE FOR: any TypeSpec\u002Ftsp change — api versions (add, bump, preview, stable, promote), resources, operations, models, properties, decorators, visibility, constraints, breaking changes, LRO, suppressions, operationId, spread model. Covers ARM resource-manager and data-plane services. DO NOT USE FOR: SDK generation, releasing SDK packages, or single MCP tool calls. INVOKES: azure-sdk-mcp:azsdk_typespec_generate_authoring_plan, azure-sdk-mcp:azsdk_run_typespec_validation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3187,3188,3189,3192],{"name":3126,"slug":3127,"type":14},{"name":11,"slug":8,"type":14},{"name":3190,"slug":3191,"type":14},"OpenAPI","openapi",{"name":3193,"slug":3194,"type":14},"Technical Writing","technical-writing","2026-07-12T08:17:39.603232",{"slug":3197,"name":3197,"fn":3198,"description":3199,"org":3200,"tags":3201,"stars":3135,"repoUrl":3136,"updatedAt":3206},"generate-sdk-locally","generate and test Azure SDKs locally","Generate, build, and test Azure SDKs locally from TypeSpec with automatic customization. WHEN: \"generate SDK locally\", \"build SDK\", \"run SDK tests\", \"run CI checks\", \"validate package\", \"run checks\", \"update changelog\", \"fix SDK build errors\", \"fix breaking changes\", \"resolve SDK generation errors\", \"customize TypeSpec\", \"rename SDK client\", \"rename SDK model\", \"hide operation from SDK\", \"fix analyzer errors\", \"resolve customization drift\", \"create subclient\", \"update metadata\", \"update version\". DO NOT USE FOR: publishing to package registries, CI pipeline configuration, API design review. INVOKES: azsdk_verify_setup, azsdk_package_generate_code, azsdk_package_build_code, azsdk_package_run_check, azsdk_package_run_tests, azsdk_customized_code_update, azsdk_package_update_changelog_content, azsdk_package_update_metadata, azsdk_package_update_version.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3202,3203,3204,3205],{"name":11,"slug":8,"type":14},{"name":3176,"slug":3177,"type":14},{"name":3147,"slug":3148,"type":14},{"name":3150,"slug":3151,"type":14},"2026-07-12T08:17:37.08523",{"slug":3208,"name":3208,"fn":3209,"description":3210,"org":3211,"tags":3212,"stars":3135,"repoUrl":3136,"updatedAt":3220},"markdown-token-optimizer","optimize markdown files for token efficiency","Analyze markdown files for token efficiency and reduce context-window bloat. **UTILITY SKILL**. DO NOT USE FOR: code optimization, general file editing, non-markdown files. TRIGGERS: optimize markdown, reduce tokens, token count, token bloat, too many tokens, make concise, shrink file, file too large, optimize for AI, token efficiency, verbose markdown, reduce file size. INVOKES: waza CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3213,3216,3219],{"name":3214,"slug":3215,"type":14},"LLM","llm",{"name":3217,"slug":3218,"type":14},"Performance","performance",{"name":3193,"slug":3194,"type":14},"2026-07-12T08:17:42.080413",{"slug":3222,"name":3222,"fn":3223,"description":3224,"org":3225,"tags":3226,"stars":3135,"repoUrl":3136,"updatedAt":3233},"pipeline-troubleshooting","troubleshoot Azure SDK CI pipelines","Diagnose and resolve failures in Azure SDK CI and generation pipelines. **UTILITY SKILL**. USE FOR: \"pipeline failed\", \"build failure\", \"CI check failing\", \"SDK generation error\", \"reproduce pipeline locally\", \"debug SDK pipeline\". DO NOT USE FOR: local build issues without pipeline context, API design review, SDK publishing. INVOKES: azure-sdk-mcp:azsdk_analyze_pipeline, azure-sdk-mcp:azsdk_package_build_code, azure-sdk-mcp:azsdk_package_run_check.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3227,3228,3229,3232],{"name":11,"slug":8,"type":14},{"name":3176,"slug":3177,"type":14},{"name":3230,"slug":3231,"type":14},"Debugging","debugging",{"name":3147,"slug":3148,"type":14},"2026-07-12T08:17:40.821512",{"slug":3235,"name":3235,"fn":3236,"description":3237,"org":3238,"tags":3239,"stars":3135,"repoUrl":3136,"updatedAt":3246},"sensei","improve skill frontmatter compliance","**WORKFLOW SKILL** — Iteratively improve skill frontmatter compliance using the Ralph loop pattern. WHEN: \"run sensei\", \"sensei help\", \"improve skill\", \"fix frontmatter\", \"skill compliance\", \"frontmatter audit\", \"score skill\", \"check skill tokens\". INVOKES: token counting tools, test runners, git commands. FOR SINGLE OPERATIONS: use token CLI directly for counts\u002Fchecks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3240,3241,3242,3245],{"name":11,"slug":8,"type":14},{"name":3089,"slug":3090,"type":14},{"name":3243,"slug":3244,"type":14},"Process Optimization","process-optimization",{"name":3193,"slug":3194,"type":14},"2026-07-12T08:17:32.970921",{"slug":3248,"name":3248,"fn":3249,"description":3250,"org":3251,"tags":3252,"stars":3135,"repoUrl":3136,"updatedAt":3258},"skill-authoring","author agent skills for agentskills.io","Write Agent Skills that comply with the agentskills.io specification. WHEN: \"create a skill\", \"new skill\", \"write a skill\", \"skill template\", \"skill structure\", \"review skill\", \"skill PR\", \"skill compliance\", \"SKILL.md format\", \"skill frontmatter\", \"skill best practices\". DO NOT USE FOR: improving existing skills (use sensei), general documentation. INVOKES: waza CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3253,3254,3257],{"name":3133,"slug":3134,"type":14},{"name":3255,"slug":3256,"type":14},"Plugin Development","plugin-development",{"name":3193,"slug":3194,"type":14},"2026-07-12T08:17:35.873862",109,{"items":3261,"total":25},[3262,3276,3288,3300,3306,3316,3329],{"slug":3263,"name":3263,"fn":3264,"description":3265,"org":3266,"tags":3267,"stars":21,"repoUrl":22,"updatedAt":3275},"agentops-agent","diagnose agent performance with AgentOps","AgentOps Doctor - surface release-readiness findings, regressions, latency spikes, error rates, and safety hits across AgentOps eval history, Azure Monitor traces, and Foundry control plane.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3268,3269,3270,3271,3272],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":3230,"slug":3231,"type":14},{"name":16,"slug":17,"type":14},{"name":3273,"slug":3274,"type":14},"Observability","observability","2026-07-12T08:19:05.564606",{"slug":1611,"name":1611,"fn":3277,"description":3278,"org":3279,"tags":3280,"stars":21,"repoUrl":22,"updatedAt":3287},"configure AgentOps release-readiness gates","Generate or update agentops.yaml (flat 1.0 schema) for AgentOps release-readiness gates. Trigger on \"configure agentops\", \"agentops.yaml\", \"set up evaluation\", \"what should I evaluate\". Infer the agent target and dataset from the codebase; ask only when nothing can be found.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3281,3282,3283,3286],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":3284,"slug":3285,"type":14},"Configuration","configuration",{"name":16,"slug":17,"type":14},"2026-07-12T08:19:06.790378",{"slug":1619,"name":1619,"fn":3289,"description":3290,"org":3291,"tags":3292,"stars":21,"repoUrl":22,"updatedAt":3299},"create datasets for AgentOps release readiness","Create or extend a small JSONL dataset for AgentOps release-readiness gates. Trigger on \"create dataset\", \"generate test data\", \"JSONL\", \"more eval rows\". Infer the agent's domain from the codebase and produce realistic rows; never fabricate data when the domain is unclear.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3293,3294,3295,3298],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":3296,"slug":3297,"type":14},"Datasets","datasets",{"name":16,"slug":17,"type":14},"2026-07-12T08:19:08.00408",{"slug":4,"name":4,"fn":5,"description":6,"org":3301,"tags":3302,"stars":21,"repoUrl":22,"updatedAt":23},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3303,3304,3305],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":16,"slug":17,"type":14},{"slug":2469,"name":2469,"fn":3307,"description":3308,"org":3309,"tags":3310,"stars":21,"repoUrl":22,"updatedAt":3315},"draft governance evidence for AI agents","Scaffold ASSERT and Red Team runners for the release gate, and draft reviewable governance evidence for ASSERT, Agent Control Specification (ACS), Guided Guardrail readiness, and red-team planning. Trigger on \"ASSERT\", \"ACS\", \"agent control\", \"guardrail\", \"red team\", \"governance\", \"release evidence\", \"scaffold assert\", \"set up red team\", \"add safety gate\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3311,3312,3313,3314],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":3089,"slug":3090,"type":14},{"name":3092,"slug":3093,"type":14},"2026-07-12T08:19:04.026045",{"slug":3317,"name":3317,"fn":3318,"description":3319,"org":3320,"tags":3321,"stars":21,"repoUrl":22,"updatedAt":3328},"agentops-report","generate and explain AgentOps release-gate reports","Read, regenerate, and explain AgentOps release-gate reports. Trigger on \"show report\", \"explain scores\", \"regenerate report\", \"what do these metrics mean\", \"where is the proof\". Operates on results.json and report.md produced by `agentops eval run`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3322,3323,3324,3325],{"name":19,"slug":20,"type":14},{"name":11,"slug":8,"type":14},{"name":3273,"slug":3274,"type":14},{"name":3326,"slug":3327,"type":14},"Reporting","reporting","2026-07-12T08:19:09.241071",{"slug":3330,"name":3330,"fn":3331,"description":3332,"org":3333,"tags":3334,"stars":21,"repoUrl":22,"updatedAt":3341},"agentops-workflow","configure AgentOps release readiness workflows","Set up AgentOps release-readiness workflows: PR eval gates, Doctor\u002Fevidence artifacts, and safe deploy handoffs to azd or Foundry prompt-agent tooling. Trigger on CI, CD, pipeline, workflow, GitHub Actions, Azure DevOps, ADO, PR gate, deploy, environments, GitFlow, release branch, promote to prod, DevOps, can we ship.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3335,3336,3337,3338],{"name":11,"slug":8,"type":14},{"name":3176,"slug":3177,"type":14},{"name":3108,"slug":3109,"type":14},{"name":3339,"slug":3340,"type":14},"GitHub Actions","github-actions","2026-07-15T06:02:51.371749"]