[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-pydantic-logfire-evals":3,"mdc-jy2qyu-key":35,"related-repo-pydantic-logfire-evals":1459,"related-org-pydantic-logfire-evals":1544},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":33,"mdContent":34},"logfire-evals","evaluate Python agent code with Logfire","Evaluate Python AI\u002Fagent code against a dataset of test cases using pydantic_evals, and review results in Logfire's Datasets & Experiments UI. Also covers redirecting an existing Braintrust Eval() suite to Logfire with no code changes. Use this skill whenever the user asks to \"set up evals\", \"add an evaluation\", \"test my agent against cases\", \"write a dataset of test cases\", \"score my LLM output\", \"add an LLM judge\", \"check tool-call correctness\", \"send Braintrust evals to Logfire\", \"migrate from Braintrust\", or mentions pydantic_evals, Braintrust, Datasets & Experiments, or evaluating AI\u002Fagent behavior against known inputs. The `pydantic_evals` workflow is Python-only; the Braintrust redirect also supports TypeScript suites, env-vars-only. Both are for scoring DEFINED test cases offline — not for instrumenting live production traffic (use `logfire-instrumentation` for that) and not for infrastructure monitoring (use `logfire-infrastructure`).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"pydantic","Pydantic","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fpydantic.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Logfire","logfire",{"name":20,"slug":21,"type":15},"Evals","evals",{"name":23,"slug":24,"type":15},"Python","python",95,"https:\u002F\u002Fgithub.com\u002Fpydantic\u002Fskills","2026-09-03T07:48:42.409632",null,3,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":28},[],"https:\u002F\u002Fgithub.com\u002Fpydantic\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Flogfire-evals","---\nname: logfire-evals\ndescription: Evaluate Python AI\u002Fagent code against a dataset of test cases using pydantic_evals, and review results in Logfire's Datasets & Experiments UI. Also covers redirecting an existing Braintrust Eval() suite to Logfire with no code changes. Use this skill whenever the user asks to \"set up evals\", \"add an evaluation\", \"test my agent against cases\", \"write a dataset of test cases\", \"score my LLM output\", \"add an LLM judge\", \"check tool-call correctness\", \"send Braintrust evals to Logfire\", \"migrate from Braintrust\", or mentions pydantic_evals, Braintrust, Datasets & Experiments, or evaluating AI\u002Fagent behavior against known inputs. The `pydantic_evals` workflow is Python-only; the Braintrust redirect also supports TypeScript suites, env-vars-only. Both are for scoring DEFINED test cases offline — not for instrumenting live production traffic (use `logfire-instrumentation` for that) and not for infrastructure monitoring (use `logfire-infrastructure`).\n---\n\n# Evaluate with pydantic_evals and Logfire\n\n## How This Works\n\n`pydantic_evals` runs your actual function or agent against a `Dataset` of `Case`s (input + expected output + metadata), scores each with one or more `Evaluator`s, and produces a report. It depends on `logfire` itself (the `datasets` extra pulls in the real SDK, not a mock), so whether `logfire.configure()` has run determines only whether results *also* upload to Logfire's Datasets & Experiments UI — omitting it keeps results entirely local and printed to the terminal, **silently, not an error**.\n\nAgentic evaluators (tool-call correctness, trajectory matching) need more than that: they read the task's own execution span tree, so without a working `logfire.configure()` they don't just fail to upload — every case reports \"No span tree available\" and the check never ran at all.\n\n## Step 1: Check for an Existing Braintrust Suite First\n\nCheap check, before anything else: does this repo already have an existing Braintrust suite — actual `Eval(...)` calls or `from braintrust import Eval` in source, not just a `braintrust` dependency listed without any real usage? This path needs **no CLI auth at all** — don't run Step 2 for it.\n\nKeep the existing `Eval()` code (Python `braintrust>=0.30.1` \u002F TypeScript `braintrust>=3.24.0` — verified versions) and redirect its next run to Logfire by changing environment variables only, no `pydantic_evals` involved:\n\n```bash\nexport BRAINTRUST_APP_URL=\"https:\u002F\u002Flogfire-us.pydantic.dev\u002Fv1\u002Fbraintrust\"  # EU: logfire-eu.pydantic.dev\nexport BRAINTRUST_API_KEY=\"\u003Clogfire-project-write-token>\"                  # Project -> Settings -> Write tokens\nunset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 \"it still hit Braintrust\" cause\n```\n\nThis is a **compatibility preview, not full parity**: covers inline\u002Fcallable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It does not cover Braintrust-hosted datasets\u002Fprompts\u002Ffunctions, BTQL, the model proxy, server-side scoring, or post-finalization feedback — and `summarize_scores=False`, a manual `flush()` without a comparison, or the Rust SDK never request the summary this endpoint needs, so nothing lands even though the run appears to succeed. Full detail and the concept-translation table (Braintrust \"project\" → Logfire dataset name, \"scorer\" → evaluator, etc.): https:\u002F\u002Fpydantic.dev\u002Fdocs\u002Flogfire\u002Fget-started\u002Fcomparisons\u002Fmigrate-from-braintrust\u002F.\n\nSkip straight to Step 5 (Verify) — the SDK's own printed result URL also opens directly in Logfire, and nothing else here (auth, dataset definition) applies to this path.\n\n**No existing Braintrust suite? Continue to Step 2 now**, before the more detailed identification in Step 3 — nothing past this point requires knowing the function\u002Fagent or dataset shape yet.\n\n## Step 2: Authenticate When the Run Needs Logfire\n\nSkip authentication and continue to Step 3 only when the user explicitly wants a local-only `pydantic_evals` run using evaluators that do not need span data; omit `logfire.configure()` in Step 4 so results stay in the terminal. Uploading results, using a hosted dataset, or running a span-based evaluator such as `ToolCorrectness` requires Logfire, so authenticate before opening or running evaluation files and target the exact project first.\n\nFor a Logfire-backed run, check first — `uvx logfire --non-interactive whoami` (JS: `npx logfire whoami`) — and skip to Step 3 if it already reports the right project and region. Otherwise, follow the full command sequence, flags, and gotchas (the `--non-interactive` requirement, why `auth` won't open a browser for you, the `LOGFIRE_TOKEN`-vs-credentials-file conflict, and token-file safety) in [Authenticate and Select the Exact Project](..\u002Flogfire-instrumentation\u002Freferences\u002Fauth.md). This CLI flow is for `logfire.configure()`; Step 3's hosted-dataset operations use a separate API key with different scopes.\n\n## Step 3: Detect What to Evaluate\n\nIdentify the function or agent under test (a PydanticAI agent, an LLM-calling function, any callable that takes an input and returns an output) and whether a dataset already exists:\n\n- **In-code dataset**: a Python module defining `Case`\u002F`Dataset` directly — the default for an agent-driven workflow.\n- **Hosted\u002Fmanaged dataset**: cases live in the Logfire UI, edited by non-engineers, pulled\u002Fpushed via a separate `LogfireAPIClient` (`from logfire.experimental.api_client import LogfireAPIClient`). `client.get_dataset(name)` with no type arguments returns a raw dict, not something `push_dataset` or `.evaluate_sync()` can take — pass the input\u002Foutput (and metadata, if used) types to get back a real `pydantic_evals.Dataset`: `client.get_dataset(name, MyInputType, MyOutputType)`. If the stored dataset contains custom evaluators, also pass their classes with `custom_evaluator_types=[ExactMatch]` (and custom report evaluators with `custom_report_evaluator_types=[...]`) so they can be deserialized. Push with `client.push_dataset(dataset)`. This needs its own API key from **Settings → API Keys** (scoped `project:read_datasets`\u002F`project:write_datasets`), not Step 2's CLI auth flow. Only relevant if the user specifically wants case editing outside code.\n\n## Step 4: Define the Dataset and Run It\n\n```bash\nuv add 'logfire[datasets]'\n```\n\n```python\nfrom dataclasses import dataclass\n\nimport logfire\nfrom pydantic_evals import Case, Dataset\nfrom pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance\n\nlogfire.configure()  # omit this and results stay local only, silently\n\n\n@dataclass\nclass ExactMatch(Evaluator[str, str]):\n    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:\n        return ctx.output == ctx.expected_output\n\n\ndef classify_sentiment(text: str) -> str:\n    ...  # the function under test\n\n\ndataset = Dataset[str, str, None](\n    name='sentiment-eval',\n    cases=[\n        Case(name='positive', inputs='I love this', expected_output='positive'),\n        Case(name='negative', inputs='This is terrible', expected_output='negative'),\n    ],\n    evaluators=[ExactMatch(), IsInstance(type_name='str')],\n)\n\nreport = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`\nreport.print(include_input=True, include_output=True)\n```\n\n**Before running the full dataset, run a smoke test on 2-3 cases** if the dataset is large or uses `LLMJudge`\u002Fany evaluator that makes a real, billed model call — a bug caught on 3 cases costs 3 model calls, the same bug caught on 300 costs 300:\n\n```python\nsmoke = Dataset(\n    name=dataset.name,\n    cases=dataset.cases[:3],\n    evaluators=dataset.evaluators,\n    report_evaluators=dataset.report_evaluators,\n)\nsmoke_report = smoke.evaluate_sync(classify_sentiment)\nsmoke_report.print(include_input=True, include_output=True)\n```\n\nConfirm the smoke run has zero unexpected errors and the assertions that should pass do. Then, if the full dataset is large or uses paid model calls, tell the user the case count and which evaluators will make model calls, and get explicit confirmation before running the full dataset — don't run an expensive full pass on the strength of a clean smoke test alone without saying so.\n\nCustom evaluators **must be `@dataclass`** subclasses — a plain class raises at run time. Case names must be unique within a dataset. The evaluators reached for most:\n\n| Evaluator | Checks |\n|-----------|--------|\n| `Equals(value)` \u002F `EqualsExpected()` | Exact match against a literal \u002F `expected_output` (no-op if `expected_output` is unset — don't rely on it silently catching that) |\n| `IsInstance(type_name)` | Output's type matches by name |\n| `LLMJudge(rubric, model=None, score=False)` | LLM-as-judge scoring; costs a real model call per case per judge — prefer boolean\u002Fcategorical rubrics over 1-10 scales (judges are unstable on continuous scores), and benchmark the judge against ~20-100 hand-labeled cases before trusting it |\n| `ToolCorrectness(expected_tools, ...)` | Which tools an agent called — reads the span tree, so needs Step 2's `logfire.configure()` to work at all, not just to upload |\n\nAlso available: `Contains`, `MaxDuration`, `TrajectoryMatch`, `ArgumentCorrectness`, `MaxToolCalls`, `MaxModelRequests` — same span-tree dependency as `ToolCorrectness` for the tool\u002Ftrajectory ones; see `pydantic_evals.evaluators` for the full set. These five agentic (span-based) evaluators need `pydantic-evals>=2.4.0` — on an older pin, check `pyproject.toml`\u002F`uv.lock` and upgrade before reaching for them, since the import itself is what fails, not a silent no-op.\n\nThe `Python` evaluator (arbitrary code execution) was removed for security reasons — don't reach for it even if an older example references it.\n\nIf editing a hosted dataset: `client.push_dataset(dataset)` **overwrites** server-side evaluators on every push, including removing ones you deleted locally — don't push a stale local copy over a dataset others have edited in the UI.\n\n## Step 5: Verify\n\nA report printing to the terminal isn't proof it reached Logfire — confirm the run actually landed. **Never report a case as passed, a score, or a run as complete without having actually checked it in this session** — if a run fails, cancels, or produces no scores, report that failure plainly; never substitute an invented score or a manual guess at what the result \"should\" be.\n\n**Came from the Step 1 Braintrust path (Step 2 skipped)?** There's no `whoami`-resolved project to look up here — use the SDK's own printed result URL instead, which already opens directly in the right Logfire project. Confirm the same things below (completion, pass mix, case detail) from that page rather than searching by name.\n\n1. **Query for the run directly, if a Logfire MCP server or API is connected** — the root span for a run is named `evaluate {name}` and carries `gen_ai.operation.name = 'experiment'`, `dataset_name`, and `task_name` attributes; find the most recent one matching your dataset's name and confirm `logfire.experiment.metadata` shows the case count and pass rate you expect. Otherwise, open **AI Evaluations → Datasets & Experiments → Experiments** in Logfire for the exact project from Step 2, and find the run by name\u002Ftimestamp.\n2. **Read the Overview tab (or the queried metadata) first**: completion count, assertion pass mix, task errors, average duration. **If completion says \"Not reported,\"** the run sent case data but never signaled it finished — treat that as a broken run, not a passing one.\n3. **Open the Cases tab**, starting from Needs Review \u002F Failed \u002F Errors, not the full list.\n4. **Drill into a failing case's trace in Live view** for the actual evidence, rather than trusting the summary score alone.\n5. **Fix and re-run** until the cases that should pass do, and any tool-call\u002Ftrajectory checks show real span data, not \"No span tree available.\"\n\nClose with a final report built from what you just confirmed — the run name, exact case count and pass rate you queried, and which evaluators ran — not a template. **Include the direct link to this experiment** (the SDK's own printed result URL, or the Datasets & Experiments page you opened it from), so the user can see the run without having to ask where to look.\n",{"data":36,"body":37},{"name":4,"description":6},{"type":38,"children":39},"root",[40,49,56,132,144,150,186,222,337,374,379,389,395,422,482,488,493,634,640,676,944,962,1032,1037,1055,1180,1267,1279,1298,1304,1316,1334,1441,1453],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"evaluate-with-pydantic_evals-and-logfire",[46],{"type":47,"value":48},"text","Evaluate with pydantic_evals and Logfire",{"type":41,"tag":50,"props":51,"children":53},"h2",{"id":52},"how-this-works",[54],{"type":47,"value":55},"How This Works",{"type":41,"tag":57,"props":58,"children":59},"p",{},[60,67,69,75,77,83,85,91,93,98,100,106,108,114,116,122,124,130],{"type":41,"tag":61,"props":62,"children":64},"code",{"className":63},[],[65],{"type":47,"value":66},"pydantic_evals",{"type":47,"value":68}," runs your actual function or agent against a ",{"type":41,"tag":61,"props":70,"children":72},{"className":71},[],[73],{"type":47,"value":74},"Dataset",{"type":47,"value":76}," of ",{"type":41,"tag":61,"props":78,"children":80},{"className":79},[],[81],{"type":47,"value":82},"Case",{"type":47,"value":84},"s (input + expected output + metadata), scores each with one or more ",{"type":41,"tag":61,"props":86,"children":88},{"className":87},[],[89],{"type":47,"value":90},"Evaluator",{"type":47,"value":92},"s, and produces a report. It depends on ",{"type":41,"tag":61,"props":94,"children":96},{"className":95},[],[97],{"type":47,"value":18},{"type":47,"value":99}," itself (the ",{"type":41,"tag":61,"props":101,"children":103},{"className":102},[],[104],{"type":47,"value":105},"datasets",{"type":47,"value":107}," extra pulls in the real SDK, not a mock), so whether ",{"type":41,"tag":61,"props":109,"children":111},{"className":110},[],[112],{"type":47,"value":113},"logfire.configure()",{"type":47,"value":115}," has run determines only whether results ",{"type":41,"tag":117,"props":118,"children":119},"em",{},[120],{"type":47,"value":121},"also",{"type":47,"value":123}," upload to Logfire's Datasets & Experiments UI — omitting it keeps results entirely local and printed to the terminal, ",{"type":41,"tag":125,"props":126,"children":127},"strong",{},[128],{"type":47,"value":129},"silently, not an error",{"type":47,"value":131},".",{"type":41,"tag":57,"props":133,"children":134},{},[135,137,142],{"type":47,"value":136},"Agentic evaluators (tool-call correctness, trajectory matching) need more than that: they read the task's own execution span tree, so without a working ",{"type":41,"tag":61,"props":138,"children":140},{"className":139},[],[141],{"type":47,"value":113},{"type":47,"value":143}," they don't just fail to upload — every case reports \"No span tree available\" and the check never ran at all.",{"type":41,"tag":50,"props":145,"children":147},{"id":146},"step-1-check-for-an-existing-braintrust-suite-first",[148],{"type":47,"value":149},"Step 1: Check for an Existing Braintrust Suite First",{"type":41,"tag":57,"props":151,"children":152},{},[153,155,161,163,169,171,177,179,184],{"type":47,"value":154},"Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual ",{"type":41,"tag":61,"props":156,"children":158},{"className":157},[],[159],{"type":47,"value":160},"Eval(...)",{"type":47,"value":162}," calls or ",{"type":41,"tag":61,"props":164,"children":166},{"className":165},[],[167],{"type":47,"value":168},"from braintrust import Eval",{"type":47,"value":170}," in source, not just a ",{"type":41,"tag":61,"props":172,"children":174},{"className":173},[],[175],{"type":47,"value":176},"braintrust",{"type":47,"value":178}," dependency listed without any real usage? This path needs ",{"type":41,"tag":125,"props":180,"children":181},{},[182],{"type":47,"value":183},"no CLI auth at all",{"type":47,"value":185}," — don't run Step 2 for it.",{"type":41,"tag":57,"props":187,"children":188},{},[189,191,197,199,205,207,213,215,220],{"type":47,"value":190},"Keep the existing ",{"type":41,"tag":61,"props":192,"children":194},{"className":193},[],[195],{"type":47,"value":196},"Eval()",{"type":47,"value":198}," code (Python ",{"type":41,"tag":61,"props":200,"children":202},{"className":201},[],[203],{"type":47,"value":204},"braintrust>=0.30.1",{"type":47,"value":206}," \u002F TypeScript ",{"type":41,"tag":61,"props":208,"children":210},{"className":209},[],[211],{"type":47,"value":212},"braintrust>=3.24.0",{"type":47,"value":214}," — verified versions) and redirect its next run to Logfire by changing environment variables only, no ",{"type":41,"tag":61,"props":216,"children":218},{"className":217},[],[219],{"type":47,"value":66},{"type":47,"value":221}," involved:",{"type":41,"tag":223,"props":224,"children":229},"pre",{"className":225,"code":226,"language":227,"meta":228,"style":228},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","export BRAINTRUST_APP_URL=\"https:\u002F\u002Flogfire-us.pydantic.dev\u002Fv1\u002Fbraintrust\"  # EU: logfire-eu.pydantic.dev\nexport BRAINTRUST_API_KEY=\"\u003Clogfire-project-write-token>\"                  # Project -> Settings -> Write tokens\nunset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 \"it still hit Braintrust\" cause\n","bash","",[230],{"type":41,"tag":61,"props":231,"children":232},{"__ignoreMap":228},[233,278,313],{"type":41,"tag":234,"props":235,"children":238},"span",{"class":236,"line":237},"line",1,[239,245,251,257,262,268,272],{"type":41,"tag":234,"props":240,"children":242},{"style":241},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[243],{"type":47,"value":244},"export",{"type":41,"tag":234,"props":246,"children":248},{"style":247},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[249],{"type":47,"value":250}," BRAINTRUST_APP_URL",{"type":41,"tag":234,"props":252,"children":254},{"style":253},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[255],{"type":47,"value":256},"=",{"type":41,"tag":234,"props":258,"children":259},{"style":253},[260],{"type":47,"value":261},"\"",{"type":41,"tag":234,"props":263,"children":265},{"style":264},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[266],{"type":47,"value":267},"https:\u002F\u002Flogfire-us.pydantic.dev\u002Fv1\u002Fbraintrust",{"type":41,"tag":234,"props":269,"children":270},{"style":253},[271],{"type":47,"value":261},{"type":41,"tag":234,"props":273,"children":275},{"style":274},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[276],{"type":47,"value":277},"  # EU: logfire-eu.pydantic.dev\n",{"type":41,"tag":234,"props":279,"children":281},{"class":236,"line":280},2,[282,286,291,295,299,304,308],{"type":41,"tag":234,"props":283,"children":284},{"style":241},[285],{"type":47,"value":244},{"type":41,"tag":234,"props":287,"children":288},{"style":247},[289],{"type":47,"value":290}," BRAINTRUST_API_KEY",{"type":41,"tag":234,"props":292,"children":293},{"style":253},[294],{"type":47,"value":256},{"type":41,"tag":234,"props":296,"children":297},{"style":253},[298],{"type":47,"value":261},{"type":41,"tag":234,"props":300,"children":301},{"style":264},[302],{"type":47,"value":303},"\u003Clogfire-project-write-token>",{"type":41,"tag":234,"props":305,"children":306},{"style":253},[307],{"type":47,"value":261},{"type":41,"tag":234,"props":309,"children":310},{"style":274},[311],{"type":47,"value":312},"                  # Project -> Settings -> Write tokens\n",{"type":41,"tag":234,"props":314,"children":315},{"class":236,"line":29},[316,322,327,332],{"type":41,"tag":234,"props":317,"children":319},{"style":318},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[320],{"type":47,"value":321},"unset",{"type":41,"tag":234,"props":323,"children":324},{"style":264},[325],{"type":47,"value":326}," BRAINTRUST_API_URL",{"type":41,"tag":234,"props":328,"children":329},{"style":264},[330],{"type":47,"value":331}," BRAINTRUST_PROXY_URL",{"type":41,"tag":234,"props":333,"children":334},{"style":274},[335],{"type":47,"value":336},"  # these override the endpoint above if set — the #1 \"it still hit Braintrust\" cause\n",{"type":41,"tag":57,"props":338,"children":339},{},[340,342,347,349,355,357,363,365,373],{"type":47,"value":341},"This is a ",{"type":41,"tag":125,"props":343,"children":344},{},[345],{"type":47,"value":346},"compatibility preview, not full parity",{"type":47,"value":348},": covers inline\u002Fcallable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It does not cover Braintrust-hosted datasets\u002Fprompts\u002Ffunctions, BTQL, the model proxy, server-side scoring, or post-finalization feedback — and ",{"type":41,"tag":61,"props":350,"children":352},{"className":351},[],[353],{"type":47,"value":354},"summarize_scores=False",{"type":47,"value":356},", a manual ",{"type":41,"tag":61,"props":358,"children":360},{"className":359},[],[361],{"type":47,"value":362},"flush()",{"type":47,"value":364}," without a comparison, or the Rust SDK never request the summary this endpoint needs, so nothing lands even though the run appears to succeed. Full detail and the concept-translation table (Braintrust \"project\" → Logfire dataset name, \"scorer\" → evaluator, etc.): ",{"type":41,"tag":366,"props":367,"children":371},"a",{"href":368,"rel":369},"https:\u002F\u002Fpydantic.dev\u002Fdocs\u002Flogfire\u002Fget-started\u002Fcomparisons\u002Fmigrate-from-braintrust\u002F",[370],"nofollow",[372],{"type":47,"value":368},{"type":47,"value":131},{"type":41,"tag":57,"props":375,"children":376},{},[377],{"type":47,"value":378},"Skip straight to Step 5 (Verify) — the SDK's own printed result URL also opens directly in Logfire, and nothing else here (auth, dataset definition) applies to this path.",{"type":41,"tag":57,"props":380,"children":381},{},[382,387],{"type":41,"tag":125,"props":383,"children":384},{},[385],{"type":47,"value":386},"No existing Braintrust suite? Continue to Step 2 now",{"type":47,"value":388},", before the more detailed identification in Step 3 — nothing past this point requires knowing the function\u002Fagent or dataset shape yet.",{"type":41,"tag":50,"props":390,"children":392},{"id":391},"step-2-authenticate-when-the-run-needs-logfire",[393],{"type":47,"value":394},"Step 2: Authenticate When the Run Needs Logfire",{"type":41,"tag":57,"props":396,"children":397},{},[398,400,405,407,412,414,420],{"type":47,"value":399},"Skip authentication and continue to Step 3 only when the user explicitly wants a local-only ",{"type":41,"tag":61,"props":401,"children":403},{"className":402},[],[404],{"type":47,"value":66},{"type":47,"value":406}," run using evaluators that do not need span data; omit ",{"type":41,"tag":61,"props":408,"children":410},{"className":409},[],[411],{"type":47,"value":113},{"type":47,"value":413}," in Step 4 so results stay in the terminal. Uploading results, using a hosted dataset, or running a span-based evaluator such as ",{"type":41,"tag":61,"props":415,"children":417},{"className":416},[],[418],{"type":47,"value":419},"ToolCorrectness",{"type":47,"value":421}," requires Logfire, so authenticate before opening or running evaluation files and target the exact project first.",{"type":41,"tag":57,"props":423,"children":424},{},[425,427,433,435,441,443,449,451,457,459,465,467,473,475,480],{"type":47,"value":426},"For a Logfire-backed run, check first — ",{"type":41,"tag":61,"props":428,"children":430},{"className":429},[],[431],{"type":47,"value":432},"uvx logfire --non-interactive whoami",{"type":47,"value":434}," (JS: ",{"type":41,"tag":61,"props":436,"children":438},{"className":437},[],[439],{"type":47,"value":440},"npx logfire whoami",{"type":47,"value":442},") — and skip to Step 3 if it already reports the right project and region. Otherwise, follow the full command sequence, flags, and gotchas (the ",{"type":41,"tag":61,"props":444,"children":446},{"className":445},[],[447],{"type":47,"value":448},"--non-interactive",{"type":47,"value":450}," requirement, why ",{"type":41,"tag":61,"props":452,"children":454},{"className":453},[],[455],{"type":47,"value":456},"auth",{"type":47,"value":458}," won't open a browser for you, the ",{"type":41,"tag":61,"props":460,"children":462},{"className":461},[],[463],{"type":47,"value":464},"LOGFIRE_TOKEN",{"type":47,"value":466},"-vs-credentials-file conflict, and token-file safety) in ",{"type":41,"tag":366,"props":468,"children":470},{"href":469},"..\u002Flogfire-instrumentation\u002Freferences\u002Fauth.md",[471],{"type":47,"value":472},"Authenticate and Select the Exact Project",{"type":47,"value":474},". This CLI flow is for ",{"type":41,"tag":61,"props":476,"children":478},{"className":477},[],[479],{"type":47,"value":113},{"type":47,"value":481},"; Step 3's hosted-dataset operations use a separate API key with different scopes.",{"type":41,"tag":50,"props":483,"children":485},{"id":484},"step-3-detect-what-to-evaluate",[486],{"type":47,"value":487},"Step 3: Detect What to Evaluate",{"type":41,"tag":57,"props":489,"children":490},{},[491],{"type":47,"value":492},"Identify the function or agent under test (a PydanticAI agent, an LLM-calling function, any callable that takes an input and returns an output) and whether a dataset already exists:",{"type":41,"tag":494,"props":495,"children":496},"ul",{},[497,522],{"type":41,"tag":498,"props":499,"children":500},"li",{},[501,506,508,513,515,520],{"type":41,"tag":125,"props":502,"children":503},{},[504],{"type":47,"value":505},"In-code dataset",{"type":47,"value":507},": a Python module defining ",{"type":41,"tag":61,"props":509,"children":511},{"className":510},[],[512],{"type":47,"value":82},{"type":47,"value":514},"\u002F",{"type":41,"tag":61,"props":516,"children":518},{"className":517},[],[519],{"type":47,"value":74},{"type":47,"value":521}," directly — the default for an agent-driven workflow.",{"type":41,"tag":498,"props":523,"children":524},{},[525,530,532,538,540,546,548,554,556,562,564,570,572,578,580,586,588,594,596,602,604,610,612,617,619,625,626,632],{"type":41,"tag":125,"props":526,"children":527},{},[528],{"type":47,"value":529},"Hosted\u002Fmanaged dataset",{"type":47,"value":531},": cases live in the Logfire UI, edited by non-engineers, pulled\u002Fpushed via a separate ",{"type":41,"tag":61,"props":533,"children":535},{"className":534},[],[536],{"type":47,"value":537},"LogfireAPIClient",{"type":47,"value":539}," (",{"type":41,"tag":61,"props":541,"children":543},{"className":542},[],[544],{"type":47,"value":545},"from logfire.experimental.api_client import LogfireAPIClient",{"type":47,"value":547},"). ",{"type":41,"tag":61,"props":549,"children":551},{"className":550},[],[552],{"type":47,"value":553},"client.get_dataset(name)",{"type":47,"value":555}," with no type arguments returns a raw dict, not something ",{"type":41,"tag":61,"props":557,"children":559},{"className":558},[],[560],{"type":47,"value":561},"push_dataset",{"type":47,"value":563}," or ",{"type":41,"tag":61,"props":565,"children":567},{"className":566},[],[568],{"type":47,"value":569},".evaluate_sync()",{"type":47,"value":571}," can take — pass the input\u002Foutput (and metadata, if used) types to get back a real ",{"type":41,"tag":61,"props":573,"children":575},{"className":574},[],[576],{"type":47,"value":577},"pydantic_evals.Dataset",{"type":47,"value":579},": ",{"type":41,"tag":61,"props":581,"children":583},{"className":582},[],[584],{"type":47,"value":585},"client.get_dataset(name, MyInputType, MyOutputType)",{"type":47,"value":587},". If the stored dataset contains custom evaluators, also pass their classes with ",{"type":41,"tag":61,"props":589,"children":591},{"className":590},[],[592],{"type":47,"value":593},"custom_evaluator_types=[ExactMatch]",{"type":47,"value":595}," (and custom report evaluators with ",{"type":41,"tag":61,"props":597,"children":599},{"className":598},[],[600],{"type":47,"value":601},"custom_report_evaluator_types=[...]",{"type":47,"value":603},") so they can be deserialized. Push with ",{"type":41,"tag":61,"props":605,"children":607},{"className":606},[],[608],{"type":47,"value":609},"client.push_dataset(dataset)",{"type":47,"value":611},". This needs its own API key from ",{"type":41,"tag":125,"props":613,"children":614},{},[615],{"type":47,"value":616},"Settings → API Keys",{"type":47,"value":618}," (scoped ",{"type":41,"tag":61,"props":620,"children":622},{"className":621},[],[623],{"type":47,"value":624},"project:read_datasets",{"type":47,"value":514},{"type":41,"tag":61,"props":627,"children":629},{"className":628},[],[630],{"type":47,"value":631},"project:write_datasets",{"type":47,"value":633},"), not Step 2's CLI auth flow. Only relevant if the user specifically wants case editing outside code.",{"type":41,"tag":50,"props":635,"children":637},{"id":636},"step-4-define-the-dataset-and-run-it",[638],{"type":47,"value":639},"Step 4: Define the Dataset and Run It",{"type":41,"tag":223,"props":641,"children":643},{"className":225,"code":642,"language":227,"meta":228,"style":228},"uv add 'logfire[datasets]'\n",[644],{"type":41,"tag":61,"props":645,"children":646},{"__ignoreMap":228},[647],{"type":41,"tag":234,"props":648,"children":649},{"class":236,"line":237},[650,656,661,666,671],{"type":41,"tag":234,"props":651,"children":653},{"style":652},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[654],{"type":47,"value":655},"uv",{"type":41,"tag":234,"props":657,"children":658},{"style":264},[659],{"type":47,"value":660}," add",{"type":41,"tag":234,"props":662,"children":663},{"style":253},[664],{"type":47,"value":665}," '",{"type":41,"tag":234,"props":667,"children":668},{"style":264},[669],{"type":47,"value":670},"logfire[datasets]",{"type":41,"tag":234,"props":672,"children":673},{"style":253},[674],{"type":47,"value":675},"'\n",{"type":41,"tag":223,"props":677,"children":680},{"className":678,"code":679,"language":24,"meta":228,"style":228},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from dataclasses import dataclass\n\nimport logfire\nfrom pydantic_evals import Case, Dataset\nfrom pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance\n\nlogfire.configure()  # omit this and results stay local only, silently\n\n\n@dataclass\nclass ExactMatch(Evaluator[str, str]):\n    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:\n        return ctx.output == ctx.expected_output\n\n\ndef classify_sentiment(text: str) -> str:\n    ...  # the function under test\n\n\ndataset = Dataset[str, str, None](\n    name='sentiment-eval',\n    cases=[\n        Case(name='positive', inputs='I love this', expected_output='positive'),\n        Case(name='negative', inputs='This is terrible', expected_output='negative'),\n    ],\n    evaluators=[ExactMatch(), IsInstance(type_name='str')],\n)\n\nreport = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`\nreport.print(include_input=True, include_output=True)\n",[681],{"type":41,"tag":61,"props":682,"children":683},{"__ignoreMap":228},[684,692,701,709,718,727,735,744,752,760,769,778,787,796,804,812,821,830,838,846,855,864,873,882,891,900,909,918,926,935],{"type":41,"tag":234,"props":685,"children":686},{"class":236,"line":237},[687],{"type":41,"tag":234,"props":688,"children":689},{},[690],{"type":47,"value":691},"from dataclasses import dataclass\n",{"type":41,"tag":234,"props":693,"children":694},{"class":236,"line":280},[695],{"type":41,"tag":234,"props":696,"children":698},{"emptyLinePlaceholder":697},true,[699],{"type":47,"value":700},"\n",{"type":41,"tag":234,"props":702,"children":703},{"class":236,"line":29},[704],{"type":41,"tag":234,"props":705,"children":706},{},[707],{"type":47,"value":708},"import logfire\n",{"type":41,"tag":234,"props":710,"children":712},{"class":236,"line":711},4,[713],{"type":41,"tag":234,"props":714,"children":715},{},[716],{"type":47,"value":717},"from pydantic_evals import Case, Dataset\n",{"type":41,"tag":234,"props":719,"children":721},{"class":236,"line":720},5,[722],{"type":41,"tag":234,"props":723,"children":724},{},[725],{"type":47,"value":726},"from pydantic_evals.evaluators import Evaluator, EvaluatorContext, IsInstance\n",{"type":41,"tag":234,"props":728,"children":730},{"class":236,"line":729},6,[731],{"type":41,"tag":234,"props":732,"children":733},{"emptyLinePlaceholder":697},[734],{"type":47,"value":700},{"type":41,"tag":234,"props":736,"children":738},{"class":236,"line":737},7,[739],{"type":41,"tag":234,"props":740,"children":741},{},[742],{"type":47,"value":743},"logfire.configure()  # omit this and results stay local only, silently\n",{"type":41,"tag":234,"props":745,"children":747},{"class":236,"line":746},8,[748],{"type":41,"tag":234,"props":749,"children":750},{"emptyLinePlaceholder":697},[751],{"type":47,"value":700},{"type":41,"tag":234,"props":753,"children":755},{"class":236,"line":754},9,[756],{"type":41,"tag":234,"props":757,"children":758},{"emptyLinePlaceholder":697},[759],{"type":47,"value":700},{"type":41,"tag":234,"props":761,"children":763},{"class":236,"line":762},10,[764],{"type":41,"tag":234,"props":765,"children":766},{},[767],{"type":47,"value":768},"@dataclass\n",{"type":41,"tag":234,"props":770,"children":772},{"class":236,"line":771},11,[773],{"type":41,"tag":234,"props":774,"children":775},{},[776],{"type":47,"value":777},"class ExactMatch(Evaluator[str, str]):\n",{"type":41,"tag":234,"props":779,"children":781},{"class":236,"line":780},12,[782],{"type":41,"tag":234,"props":783,"children":784},{},[785],{"type":47,"value":786},"    def evaluate(self, ctx: EvaluatorContext[str, str]) -> bool:\n",{"type":41,"tag":234,"props":788,"children":790},{"class":236,"line":789},13,[791],{"type":41,"tag":234,"props":792,"children":793},{},[794],{"type":47,"value":795},"        return ctx.output == ctx.expected_output\n",{"type":41,"tag":234,"props":797,"children":799},{"class":236,"line":798},14,[800],{"type":41,"tag":234,"props":801,"children":802},{"emptyLinePlaceholder":697},[803],{"type":47,"value":700},{"type":41,"tag":234,"props":805,"children":807},{"class":236,"line":806},15,[808],{"type":41,"tag":234,"props":809,"children":810},{"emptyLinePlaceholder":697},[811],{"type":47,"value":700},{"type":41,"tag":234,"props":813,"children":815},{"class":236,"line":814},16,[816],{"type":41,"tag":234,"props":817,"children":818},{},[819],{"type":47,"value":820},"def classify_sentiment(text: str) -> str:\n",{"type":41,"tag":234,"props":822,"children":824},{"class":236,"line":823},17,[825],{"type":41,"tag":234,"props":826,"children":827},{},[828],{"type":47,"value":829},"    ...  # the function under test\n",{"type":41,"tag":234,"props":831,"children":833},{"class":236,"line":832},18,[834],{"type":41,"tag":234,"props":835,"children":836},{"emptyLinePlaceholder":697},[837],{"type":47,"value":700},{"type":41,"tag":234,"props":839,"children":841},{"class":236,"line":840},19,[842],{"type":41,"tag":234,"props":843,"children":844},{"emptyLinePlaceholder":697},[845],{"type":47,"value":700},{"type":41,"tag":234,"props":847,"children":849},{"class":236,"line":848},20,[850],{"type":41,"tag":234,"props":851,"children":852},{},[853],{"type":47,"value":854},"dataset = Dataset[str, str, None](\n",{"type":41,"tag":234,"props":856,"children":858},{"class":236,"line":857},21,[859],{"type":41,"tag":234,"props":860,"children":861},{},[862],{"type":47,"value":863},"    name='sentiment-eval',\n",{"type":41,"tag":234,"props":865,"children":867},{"class":236,"line":866},22,[868],{"type":41,"tag":234,"props":869,"children":870},{},[871],{"type":47,"value":872},"    cases=[\n",{"type":41,"tag":234,"props":874,"children":876},{"class":236,"line":875},23,[877],{"type":41,"tag":234,"props":878,"children":879},{},[880],{"type":47,"value":881},"        Case(name='positive', inputs='I love this', expected_output='positive'),\n",{"type":41,"tag":234,"props":883,"children":885},{"class":236,"line":884},24,[886],{"type":41,"tag":234,"props":887,"children":888},{},[889],{"type":47,"value":890},"        Case(name='negative', inputs='This is terrible', expected_output='negative'),\n",{"type":41,"tag":234,"props":892,"children":894},{"class":236,"line":893},25,[895],{"type":41,"tag":234,"props":896,"children":897},{},[898],{"type":47,"value":899},"    ],\n",{"type":41,"tag":234,"props":901,"children":903},{"class":236,"line":902},26,[904],{"type":41,"tag":234,"props":905,"children":906},{},[907],{"type":47,"value":908},"    evaluators=[ExactMatch(), IsInstance(type_name='str')],\n",{"type":41,"tag":234,"props":910,"children":912},{"class":236,"line":911},27,[913],{"type":41,"tag":234,"props":914,"children":915},{},[916],{"type":47,"value":917},")\n",{"type":41,"tag":234,"props":919,"children":921},{"class":236,"line":920},28,[922],{"type":41,"tag":234,"props":923,"children":924},{"emptyLinePlaceholder":697},[925],{"type":47,"value":700},{"type":41,"tag":234,"props":927,"children":929},{"class":236,"line":928},29,[930],{"type":41,"tag":234,"props":931,"children":932},{},[933],{"type":47,"value":934},"report = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`\n",{"type":41,"tag":234,"props":936,"children":938},{"class":236,"line":937},30,[939],{"type":41,"tag":234,"props":940,"children":941},{},[942],{"type":47,"value":943},"report.print(include_input=True, include_output=True)\n",{"type":41,"tag":57,"props":945,"children":946},{},[947,952,954,960],{"type":41,"tag":125,"props":948,"children":949},{},[950],{"type":47,"value":951},"Before running the full dataset, run a smoke test on 2-3 cases",{"type":47,"value":953}," if the dataset is large or uses ",{"type":41,"tag":61,"props":955,"children":957},{"className":956},[],[958],{"type":47,"value":959},"LLMJudge",{"type":47,"value":961},"\u002Fany evaluator that makes a real, billed model call — a bug caught on 3 cases costs 3 model calls, the same bug caught on 300 costs 300:",{"type":41,"tag":223,"props":963,"children":965},{"className":678,"code":964,"language":24,"meta":228,"style":228},"smoke = Dataset(\n    name=dataset.name,\n    cases=dataset.cases[:3],\n    evaluators=dataset.evaluators,\n    report_evaluators=dataset.report_evaluators,\n)\nsmoke_report = smoke.evaluate_sync(classify_sentiment)\nsmoke_report.print(include_input=True, include_output=True)\n",[966],{"type":41,"tag":61,"props":967,"children":968},{"__ignoreMap":228},[969,977,985,993,1001,1009,1016,1024],{"type":41,"tag":234,"props":970,"children":971},{"class":236,"line":237},[972],{"type":41,"tag":234,"props":973,"children":974},{},[975],{"type":47,"value":976},"smoke = Dataset(\n",{"type":41,"tag":234,"props":978,"children":979},{"class":236,"line":280},[980],{"type":41,"tag":234,"props":981,"children":982},{},[983],{"type":47,"value":984},"    name=dataset.name,\n",{"type":41,"tag":234,"props":986,"children":987},{"class":236,"line":29},[988],{"type":41,"tag":234,"props":989,"children":990},{},[991],{"type":47,"value":992},"    cases=dataset.cases[:3],\n",{"type":41,"tag":234,"props":994,"children":995},{"class":236,"line":711},[996],{"type":41,"tag":234,"props":997,"children":998},{},[999],{"type":47,"value":1000},"    evaluators=dataset.evaluators,\n",{"type":41,"tag":234,"props":1002,"children":1003},{"class":236,"line":720},[1004],{"type":41,"tag":234,"props":1005,"children":1006},{},[1007],{"type":47,"value":1008},"    report_evaluators=dataset.report_evaluators,\n",{"type":41,"tag":234,"props":1010,"children":1011},{"class":236,"line":729},[1012],{"type":41,"tag":234,"props":1013,"children":1014},{},[1015],{"type":47,"value":917},{"type":41,"tag":234,"props":1017,"children":1018},{"class":236,"line":737},[1019],{"type":41,"tag":234,"props":1020,"children":1021},{},[1022],{"type":47,"value":1023},"smoke_report = smoke.evaluate_sync(classify_sentiment)\n",{"type":41,"tag":234,"props":1025,"children":1026},{"class":236,"line":746},[1027],{"type":41,"tag":234,"props":1028,"children":1029},{},[1030],{"type":47,"value":1031},"smoke_report.print(include_input=True, include_output=True)\n",{"type":41,"tag":57,"props":1033,"children":1034},{},[1035],{"type":47,"value":1036},"Confirm the smoke run has zero unexpected errors and the assertions that should pass do. Then, if the full dataset is large or uses paid model calls, tell the user the case count and which evaluators will make model calls, and get explicit confirmation before running the full dataset — don't run an expensive full pass on the strength of a clean smoke test alone without saying so.",{"type":41,"tag":57,"props":1038,"children":1039},{},[1040,1042,1053],{"type":47,"value":1041},"Custom evaluators ",{"type":41,"tag":125,"props":1043,"children":1044},{},[1045,1047],{"type":47,"value":1046},"must be ",{"type":41,"tag":61,"props":1048,"children":1050},{"className":1049},[],[1051],{"type":47,"value":1052},"@dataclass",{"type":47,"value":1054}," subclasses — a plain class raises at run time. Case names must be unique within a dataset. The evaluators reached for most:",{"type":41,"tag":1056,"props":1057,"children":1058},"table",{},[1059,1077],{"type":41,"tag":1060,"props":1061,"children":1062},"thead",{},[1063],{"type":41,"tag":1064,"props":1065,"children":1066},"tr",{},[1067,1072],{"type":41,"tag":1068,"props":1069,"children":1070},"th",{},[1071],{"type":47,"value":90},{"type":41,"tag":1068,"props":1073,"children":1074},{},[1075],{"type":47,"value":1076},"Checks",{"type":41,"tag":1078,"props":1079,"children":1080},"tbody",{},[1081,1122,1139,1156],{"type":41,"tag":1064,"props":1082,"children":1083},{},[1084,1102],{"type":41,"tag":1085,"props":1086,"children":1087},"td",{},[1088,1094,1096],{"type":41,"tag":61,"props":1089,"children":1091},{"className":1090},[],[1092],{"type":47,"value":1093},"Equals(value)",{"type":47,"value":1095}," \u002F ",{"type":41,"tag":61,"props":1097,"children":1099},{"className":1098},[],[1100],{"type":47,"value":1101},"EqualsExpected()",{"type":41,"tag":1085,"props":1103,"children":1104},{},[1105,1107,1113,1115,1120],{"type":47,"value":1106},"Exact match against a literal \u002F ",{"type":41,"tag":61,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":47,"value":1112},"expected_output",{"type":47,"value":1114}," (no-op if ",{"type":41,"tag":61,"props":1116,"children":1118},{"className":1117},[],[1119],{"type":47,"value":1112},{"type":47,"value":1121}," is unset — don't rely on it silently catching that)",{"type":41,"tag":1064,"props":1123,"children":1124},{},[1125,1134],{"type":41,"tag":1085,"props":1126,"children":1127},{},[1128],{"type":41,"tag":61,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":47,"value":1133},"IsInstance(type_name)",{"type":41,"tag":1085,"props":1135,"children":1136},{},[1137],{"type":47,"value":1138},"Output's type matches by name",{"type":41,"tag":1064,"props":1140,"children":1141},{},[1142,1151],{"type":41,"tag":1085,"props":1143,"children":1144},{},[1145],{"type":41,"tag":61,"props":1146,"children":1148},{"className":1147},[],[1149],{"type":47,"value":1150},"LLMJudge(rubric, model=None, score=False)",{"type":41,"tag":1085,"props":1152,"children":1153},{},[1154],{"type":47,"value":1155},"LLM-as-judge scoring; costs a real model call per case per judge — prefer boolean\u002Fcategorical rubrics over 1-10 scales (judges are unstable on continuous scores), and benchmark the judge against ~20-100 hand-labeled cases before trusting it",{"type":41,"tag":1064,"props":1157,"children":1158},{},[1159,1168],{"type":41,"tag":1085,"props":1160,"children":1161},{},[1162],{"type":41,"tag":61,"props":1163,"children":1165},{"className":1164},[],[1166],{"type":47,"value":1167},"ToolCorrectness(expected_tools, ...)",{"type":41,"tag":1085,"props":1169,"children":1170},{},[1171,1173,1178],{"type":47,"value":1172},"Which tools an agent called — reads the span tree, so needs Step 2's ",{"type":41,"tag":61,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":47,"value":113},{"type":47,"value":1179}," to work at all, not just to upload",{"type":41,"tag":57,"props":1181,"children":1182},{},[1183,1185,1191,1193,1199,1200,1206,1207,1213,1214,1220,1221,1227,1229,1234,1236,1242,1244,1250,1252,1258,1259,1265],{"type":47,"value":1184},"Also available: ",{"type":41,"tag":61,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":47,"value":1190},"Contains",{"type":47,"value":1192},", ",{"type":41,"tag":61,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":47,"value":1198},"MaxDuration",{"type":47,"value":1192},{"type":41,"tag":61,"props":1201,"children":1203},{"className":1202},[],[1204],{"type":47,"value":1205},"TrajectoryMatch",{"type":47,"value":1192},{"type":41,"tag":61,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":47,"value":1212},"ArgumentCorrectness",{"type":47,"value":1192},{"type":41,"tag":61,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":47,"value":1219},"MaxToolCalls",{"type":47,"value":1192},{"type":41,"tag":61,"props":1222,"children":1224},{"className":1223},[],[1225],{"type":47,"value":1226},"MaxModelRequests",{"type":47,"value":1228}," — same span-tree dependency as ",{"type":41,"tag":61,"props":1230,"children":1232},{"className":1231},[],[1233],{"type":47,"value":419},{"type":47,"value":1235}," for the tool\u002Ftrajectory ones; see ",{"type":41,"tag":61,"props":1237,"children":1239},{"className":1238},[],[1240],{"type":47,"value":1241},"pydantic_evals.evaluators",{"type":47,"value":1243}," for the full set. These five agentic (span-based) evaluators need ",{"type":41,"tag":61,"props":1245,"children":1247},{"className":1246},[],[1248],{"type":47,"value":1249},"pydantic-evals>=2.4.0",{"type":47,"value":1251}," — on an older pin, check ",{"type":41,"tag":61,"props":1253,"children":1255},{"className":1254},[],[1256],{"type":47,"value":1257},"pyproject.toml",{"type":47,"value":514},{"type":41,"tag":61,"props":1260,"children":1262},{"className":1261},[],[1263],{"type":47,"value":1264},"uv.lock",{"type":47,"value":1266}," and upgrade before reaching for them, since the import itself is what fails, not a silent no-op.",{"type":41,"tag":57,"props":1268,"children":1269},{},[1270,1272,1277],{"type":47,"value":1271},"The ",{"type":41,"tag":61,"props":1273,"children":1275},{"className":1274},[],[1276],{"type":47,"value":23},{"type":47,"value":1278}," evaluator (arbitrary code execution) was removed for security reasons — don't reach for it even if an older example references it.",{"type":41,"tag":57,"props":1280,"children":1281},{},[1282,1284,1289,1291,1296],{"type":47,"value":1283},"If editing a hosted dataset: ",{"type":41,"tag":61,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":47,"value":609},{"type":47,"value":1290}," ",{"type":41,"tag":125,"props":1292,"children":1293},{},[1294],{"type":47,"value":1295},"overwrites",{"type":47,"value":1297}," server-side evaluators on every push, including removing ones you deleted locally — don't push a stale local copy over a dataset others have edited in the UI.",{"type":41,"tag":50,"props":1299,"children":1301},{"id":1300},"step-5-verify",[1302],{"type":47,"value":1303},"Step 5: Verify",{"type":41,"tag":57,"props":1305,"children":1306},{},[1307,1309,1314],{"type":47,"value":1308},"A report printing to the terminal isn't proof it reached Logfire — confirm the run actually landed. ",{"type":41,"tag":125,"props":1310,"children":1311},{},[1312],{"type":47,"value":1313},"Never report a case as passed, a score, or a run as complete without having actually checked it in this session",{"type":47,"value":1315}," — if a run fails, cancels, or produces no scores, report that failure plainly; never substitute an invented score or a manual guess at what the result \"should\" be.",{"type":41,"tag":57,"props":1317,"children":1318},{},[1319,1324,1326,1332],{"type":41,"tag":125,"props":1320,"children":1321},{},[1322],{"type":47,"value":1323},"Came from the Step 1 Braintrust path (Step 2 skipped)?",{"type":47,"value":1325}," There's no ",{"type":41,"tag":61,"props":1327,"children":1329},{"className":1328},[],[1330],{"type":47,"value":1331},"whoami",{"type":47,"value":1333},"-resolved project to look up here — use the SDK's own printed result URL instead, which already opens directly in the right Logfire project. Confirm the same things below (completion, pass mix, case detail) from that page rather than searching by name.",{"type":41,"tag":1335,"props":1336,"children":1337},"ol",{},[1338,1394,1411,1421,1431],{"type":41,"tag":498,"props":1339,"children":1340},{},[1341,1346,1348,1354,1356,1362,1363,1369,1371,1377,1379,1385,1387,1392],{"type":41,"tag":125,"props":1342,"children":1343},{},[1344],{"type":47,"value":1345},"Query for the run directly, if a Logfire MCP server or API is connected",{"type":47,"value":1347}," — the root span for a run is named ",{"type":41,"tag":61,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":47,"value":1353},"evaluate {name}",{"type":47,"value":1355}," and carries ",{"type":41,"tag":61,"props":1357,"children":1359},{"className":1358},[],[1360],{"type":47,"value":1361},"gen_ai.operation.name = 'experiment'",{"type":47,"value":1192},{"type":41,"tag":61,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":47,"value":1368},"dataset_name",{"type":47,"value":1370},", and ",{"type":41,"tag":61,"props":1372,"children":1374},{"className":1373},[],[1375],{"type":47,"value":1376},"task_name",{"type":47,"value":1378}," attributes; find the most recent one matching your dataset's name and confirm ",{"type":41,"tag":61,"props":1380,"children":1382},{"className":1381},[],[1383],{"type":47,"value":1384},"logfire.experiment.metadata",{"type":47,"value":1386}," shows the case count and pass rate you expect. Otherwise, open ",{"type":41,"tag":125,"props":1388,"children":1389},{},[1390],{"type":47,"value":1391},"AI Evaluations → Datasets & Experiments → Experiments",{"type":47,"value":1393}," in Logfire for the exact project from Step 2, and find the run by name\u002Ftimestamp.",{"type":41,"tag":498,"props":1395,"children":1396},{},[1397,1402,1404,1409],{"type":41,"tag":125,"props":1398,"children":1399},{},[1400],{"type":47,"value":1401},"Read the Overview tab (or the queried metadata) first",{"type":47,"value":1403},": completion count, assertion pass mix, task errors, average duration. ",{"type":41,"tag":125,"props":1405,"children":1406},{},[1407],{"type":47,"value":1408},"If completion says \"Not reported,\"",{"type":47,"value":1410}," the run sent case data but never signaled it finished — treat that as a broken run, not a passing one.",{"type":41,"tag":498,"props":1412,"children":1413},{},[1414,1419],{"type":41,"tag":125,"props":1415,"children":1416},{},[1417],{"type":47,"value":1418},"Open the Cases tab",{"type":47,"value":1420},", starting from Needs Review \u002F Failed \u002F Errors, not the full list.",{"type":41,"tag":498,"props":1422,"children":1423},{},[1424,1429],{"type":41,"tag":125,"props":1425,"children":1426},{},[1427],{"type":47,"value":1428},"Drill into a failing case's trace in Live view",{"type":47,"value":1430}," for the actual evidence, rather than trusting the summary score alone.",{"type":41,"tag":498,"props":1432,"children":1433},{},[1434,1439],{"type":41,"tag":125,"props":1435,"children":1436},{},[1437],{"type":47,"value":1438},"Fix and re-run",{"type":47,"value":1440}," until the cases that should pass do, and any tool-call\u002Ftrajectory checks show real span data, not \"No span tree available.\"",{"type":41,"tag":57,"props":1442,"children":1443},{},[1444,1446,1451],{"type":47,"value":1445},"Close with a final report built from what you just confirmed — the run name, exact case count and pass rate you queried, and which evaluators ran — not a template. ",{"type":41,"tag":125,"props":1447,"children":1448},{},[1449],{"type":47,"value":1450},"Include the direct link to this experiment",{"type":47,"value":1452}," (the SDK's own printed result URL, or the Datasets & Experiments page you opened it from), so the user can see the run without having to ask where to look.",{"type":41,"tag":1454,"props":1455,"children":1456},"style",{},[1457],{"type":47,"value":1458},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1460,"total":754},[1461,1476,1483,1498,1511,1524,1534],{"slug":1462,"name":1462,"fn":1463,"description":1464,"org":1465,"tags":1466,"stars":25,"repoUrl":26,"updatedAt":1475},"building-pydantic-ai-agents","build AI agents with Pydantic AI","Build AI agents with Pydantic AI — tools, capabilities (including on-demand loading), structured output, streaming, testing, and multi-agent patterns. Use when the user mentions Pydantic AI, imports pydantic_ai, or asks to build an AI agent, add tools\u002Fcapabilities, defer capability loading, stream output, define agents from YAML, or test agent behavior.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1467,1470,1473,1474],{"name":1468,"slug":1469,"type":15},"Agents","agents",{"name":1471,"slug":1472,"type":15},"LLM","llm",{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},"2026-08-05T05:36:25.389247",{"slug":4,"name":4,"fn":5,"description":6,"org":1477,"tags":1478,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1479,1480,1481,1482],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"slug":1484,"name":1484,"fn":1485,"description":1486,"org":1487,"tags":1488,"stars":25,"repoUrl":26,"updatedAt":1497},"logfire-infrastructure","monitor infrastructure metrics with Logfire","Monitor hosts, Docker containers, Kubernetes clusters, database\u002Fqueue\u002Fcache servers, and cloud-provider metrics with Pydantic Logfire — no application code required. Use this skill whenever the user asks to \"monitor my host\u002Fserver\u002FVM\", \"monitor my Docker containers\", \"monitor my Kubernetes cluster\", \"send infrastructure metrics to Logfire\", \"watch my database\u002FPostgres\u002FRedis\u002FMongoDB\u002FKafka\", \"collect cloud metrics\" (AWS\u002FGCP), or mentions the OpenTelemetry Collector in the context of Logfire. This is infrastructure only — for instrumenting APPLICATION CODE (traces, logs, AI\u002Fagent spans) use the logfire-instrumentation skill instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1489,1492,1493,1496],{"name":1490,"slug":1491,"type":15},"Infrastructure","infrastructure",{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-09-03T07:48:50.066951",{"slug":1499,"name":1499,"fn":1500,"description":1501,"org":1502,"tags":1503,"stars":25,"repoUrl":26,"updatedAt":1510},"logfire-instrumentation","instrument applications with Pydantic Logfire","Add Pydantic Logfire observability to application code — traces, logs, metrics, and AI\u002Fagent spans. Use when the user asks to add or configure Logfire, observability, tracing, logging, or monitoring; maximize useful telemetry; or understand what an app is doing. Supports Python, JavaScript\u002FTypeScript, Rust, and major AI agent frameworks including Pydantic AI, OpenAI Agents SDK, Claude Agent SDK, LangChain, LangGraph, CrewAI, AutoGen, and Google ADK. For infrastructure-only monitoring (hosts, Docker, Kubernetes, databases, or cloud metrics with no app-code changes), use `logfire-infrastructure`. For evaluating AI\u002Fagent behavior against test datasets, use `logfire-evals`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1504,1507,1508,1509],{"name":1505,"slug":1506,"type":15},"Engineering","engineering",{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},"2026-09-03T07:26:53.684135",{"slug":1512,"name":1512,"fn":1513,"description":1514,"org":1515,"tags":1516,"stars":25,"repoUrl":26,"updatedAt":1523},"logfire-query","query and analyze Logfire telemetry","Query and analyze Logfire telemetry data — traces, logs, spans, metrics, summaries, and SQL results. Use this skill when the user asks to \"query logfire\", \"search traces\", \"find logs\", \"query data\", \"search spans\", \"look up errors in logfire\", \"get metrics from logfire\", \"analyze telemetry\", \"summarize errors\", \"find root cause\", or add Logfire querying capabilities to code. Do not use this skill for direct Logfire UI, browser, live-view, Explore-page, or link-opening requests; use logfire-ui instead. If \"show\" or \"view\" wording is ambiguous, ask whether the user wants a UI view or query analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1517,1520,1521,1522],{"name":1518,"slug":1519,"type":15},"Data Analysis","data-analysis",{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},"2026-07-16T05:59:18.201951",{"slug":1525,"name":1525,"fn":1526,"description":1527,"org":1528,"tags":1529,"stars":25,"repoUrl":26,"updatedAt":1533},"logfire-setup","set up Pydantic Logfire observability","Entry point for Pydantic Logfire — an observability, monitoring, and evals platform. Use this skill when the user asks to \"set up Logfire\", \"add Logfire to my project\", \"get me set up properly with Logfire\", \"send as much data as would be useful\", mentions Logfire without a specific scope, or their request spans more than one of instrumenting application code \u002F monitoring infrastructure \u002F evaluating AI behavior. If the request is clearly scoped to exactly one of those, fetch that specific skill directly instead of this one — this skill exists to route, not to duplicate their content.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1530,1531,1532],{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},"2026-09-03T07:48:43.034559",{"slug":1535,"name":1535,"fn":1536,"description":1537,"org":1538,"tags":1539,"stars":25,"repoUrl":26,"updatedAt":1543},"logfire-ui","navigate Logfire project pages","Open or return Logfire project pages, live views, trace links, and Explore pages in the Codex browser without querying telemetry first. Use this skill when the user asks to \"open in Logfire\", \"show in the live view\", \"open Explore\", \"open the UI\", \"show in Codex\", \"use the browser\", \"give me a link\", or asks for a Logfire GUI\u002Fbrowser\u002Flive-view presentation of a project, time range, service, span, trace, log, or filter. If \"show\" or \"view\" wording is ambiguous, ask whether the user wants a UI view or query analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1540,1541,1542],{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},"2026-07-16T05:59:17.66486",{"items":1545,"total":754},[1546,1553,1560,1567,1574,1581,1587,1593,1607],{"slug":1462,"name":1462,"fn":1463,"description":1464,"org":1547,"tags":1548,"stars":25,"repoUrl":26,"updatedAt":1475},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1549,1550,1551,1552],{"name":1468,"slug":1469,"type":15},{"name":1471,"slug":1472,"type":15},{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":1554,"tags":1555,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1556,1557,1558,1559],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"slug":1484,"name":1484,"fn":1485,"description":1486,"org":1561,"tags":1562,"stars":25,"repoUrl":26,"updatedAt":1497},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1563,1564,1565,1566],{"name":1490,"slug":1491,"type":15},{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},{"slug":1499,"name":1499,"fn":1500,"description":1501,"org":1568,"tags":1569,"stars":25,"repoUrl":26,"updatedAt":1510},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1570,1571,1572,1573],{"name":1505,"slug":1506,"type":15},{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},{"slug":1512,"name":1512,"fn":1513,"description":1514,"org":1575,"tags":1576,"stars":25,"repoUrl":26,"updatedAt":1523},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1577,1578,1579,1580],{"name":1518,"slug":1519,"type":15},{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},{"slug":1525,"name":1525,"fn":1526,"description":1527,"org":1582,"tags":1583,"stars":25,"repoUrl":26,"updatedAt":1533},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1584,1585,1586],{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},{"slug":1535,"name":1535,"fn":1536,"description":1537,"org":1588,"tags":1589,"stars":25,"repoUrl":26,"updatedAt":1543},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1590,1591,1592],{"name":17,"slug":18,"type":15},{"name":1494,"slug":1495,"type":15},{"name":13,"slug":14,"type":15},{"slug":8,"name":8,"fn":1594,"description":1595,"org":1596,"tags":1597,"stars":25,"repoUrl":26,"updatedAt":1606},"model and validate data with Pydantic","Pydantic is a Python data validation and serialization library, based on type hints. Use this skill whenever you need to do relatively complex data modeling using Pydantic, e.g. when adding constraints, defining a model hierarchy with subclasses, etc.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1598,1601,1602,1603],{"name":1599,"slug":1600,"type":15},"Data Modeling","data-modeling",{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},{"name":1604,"slug":1605,"type":15},"Validation","validation","2026-07-23T06:05:52.745966",{"slug":1608,"name":1608,"fn":1609,"description":1610,"org":1611,"tags":1612,"stars":25,"repoUrl":26,"updatedAt":1619},"pydantic-ai-harness","extend Pydantic AI agents with capabilities","Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- Code Mode (collapse many tool calls into one sandboxed Python execution), a filesystem and shell, sub-agents, planning, context compaction, and more. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want first-party filesystem\u002Fshell\u002Fsub-agent\u002Fplanning\u002Fcompaction capabilities for a Pydantic AI agent, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1613,1614,1617,1618],{"name":1468,"slug":1469,"type":15},{"name":1615,"slug":1616,"type":15},"Code Execution","code-execution",{"name":9,"slug":8,"type":15},{"name":23,"slug":24,"type":15},"2026-07-30T05:28:54.443013"]