Skill

foundry-evaluations

Name: foundry-evaluations
Author: Azure (Microsoft)

evaluate agent quality with Foundry Evals

Published by Azure (Microsoft) Updated Jul 12

Covers Azure LLM Evals OpenAI

Description

Evaluate agent quality using Foundry OpenAI Evals API. Create evaluations, run them against models, and analyze results.

SKILL.md

Foundry Evaluations — OpenAI Evals API

You can evaluate agent and model quality using the Foundry OpenAI Evals API. Create evaluation definitions with testing criteria, run them against models, and analyze pass/fail results.

Endpoint

All requests: http://localhost:8443 with ?api-version=2025-11-15-preview. Auth is automatic.

Operations

List evaluations

curl -s 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview'

Create an evaluation

curl -s -X POST 'http://localhost:8443/openai/evals?api-version=2025-11-15-preview' \
  -H 'Content-Type: application/json' \
  -d '{"name":"quality-check","data_source_config":{"type":"custom","item_schema":{"type":"object","properties":{"input":{"type":"string"},"expected":{"type":"string"}},"required":["input","expected"]}},"testing_criteria":[{"type":"string_check","name":"exact-match","input":"{{sample.output_text}}","reference":"{{item.expected}}","operation":"eq"}]}'

Run an evaluation

curl -s -X POST 'http://localhost:8443/openai/evals/eval_abc123/runs?api-version=2025-11-15-preview' \
  -H 'Content-Type: application/json' \
  -d '{"name":"run-1","data_source":{"type":"jsonl","source":{"type":"file_content","content":[{"item":{"input":"2+2","expected":"4"}}]}}}'

List evaluators (built-in + custom)

curl -s 'http://localhost:8443/evaluators?api-version=2025-11-15-preview'

List evaluation rules

curl -s 'http://localhost:8443/evaluationrules?api-version=2025-11-15-preview'

When to use

Measuring agent quality (accuracy, safety, groundedness)
A/B testing model versions or prompt changes
Continuous evaluation of production agent responses
Red-teaming and safety testing

When NOT to use

For runtime inference (use /v1/chat/completions or /openai/responses)
For conversation management (use foundry-conversations skill)

More skills from the kars repository

View all 11 skills

SkillTagsUpdated

More from Azure (Microsoft)

View publisher

SkillTagsUpdated