[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-langchain-langsmith-evaluator":3,"mdc--eaolfm-key":33,"related-repo-langchain-langsmith-evaluator":2068,"related-org-langchain-langsmith-evaluator":2103},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"langsmith-evaluator","build LangSmith evaluation pipelines","INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"langchain","LangChain","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Flangchain.png","langchain-ai",[13,17,20],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"LangSmith","langsmith",139,"https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Flangsmith-skills","2026-04-11T04:40:28.078348",null,17,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Flangsmith-skills\u002Ftree\u002FHEAD\u002Fconfig\u002Fskills\u002Flangsmith-evaluator","---\nname: langsmith-evaluator\ndescription: \"INVOKE THIS SKILL when building evaluation pipelines for LangSmith. Covers three core components: (1) Creating Evaluators - LLM-as-Judge, custom code; (2) Defining Run Functions - how to capture outputs and trajectories from your agent; (3) Running Evaluations - locally with evaluate() or auto-run via LangSmith. Uses the langsmith CLI tool.\"\n---\n\n\u003Coneliner>\nThree core components: **(1) Creating Evaluators** - LLM-as-Judge, custom code; **(2) Defining Run Functions** - capture agent outputs\u002Ftrajectories for evaluation; **(3) Running Evaluations** - locally with `evaluate()` or auto-run via uploaded evaluators. Python and TypeScript examples included.\n\u003C\u002Foneliner>\n\n\u003Csetup>\nEnvironment Variables\n\n```bash\nLANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED\nLANGSMITH_PROJECT=your-project-name                   # Check this to know which project has traces\nLANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys\nOPENAI_API_KEY=your_openai_key                        # For LLM as Judge\n```\n\nAuthentication is REQUIRED: either set the `LANGSMITH_API_KEY` environment variable, or pass the `--api-key` flag to CLI commands (preferred):\n```bash\nlangsmith evaluator list --api-key $LANGSMITH_API_KEY\n```\n\n**IMPORTANT:** Always check the environment variables or `.env` file for `LANGSMITH_PROJECT` before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.\n\nPython Dependencies\n```bash\npip install langsmith langchain-openai python-dotenv\n```\n\nCLI Tool (for uploading evaluators)\n```bash\ncurl -sSL https:\u002F\u002Fraw.githubusercontent.com\u002Flangchain-ai\u002Flangsmith-cli\u002Fmain\u002Fscripts\u002Finstall.sh | sh\n```\n\nJavaScript Dependencies\n```bash\nnpm install langsmith openai\n```\n\u003C\u002Fsetup>\n\n\u003Ccrucial_requirement>\n## Golden Rule: Inspect Before You Implement\n\n**CRITICAL:** Before writing ANY evaluator or extraction logic, you MUST:\n1. **Run your agent** on sample inputs and capture the actual output\n2. **Inspect the output** - print it, query LangSmith traces, understand the exact structure\n3. **Only then** write code that processes that output\n\nOutput structures vary significantly by framework, agent type, and configuration. Never assume the shape - always verify first. Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution.\n\u003C\u002Fcrucial_requirement>\n\n\u003Cevaluator_format>\n## Offline vs Online Evaluators\n\n**Offline Evaluators** (attached to datasets):\n- Function signature: `(run, example)` - receives both run outputs and dataset example\n- Use case: Comparing agent outputs to expected values in a dataset\n- Upload with: `--dataset \"Dataset Name\"`\n\n**Online Evaluators** (attached to projects):\n- Function signature: `(run)` - receives only run outputs, NO example parameter\n- Use case: Real-time quality checks on production runs (no reference data)\n- Upload with: `--project \"Project Name\"`\n\n**CRITICAL - Return Format:**\n- Each evaluator returns **ONE metric only**. For multiple metrics, create multiple evaluator functions.\n- Do NOT return `{\"metric_name\": value}` or lists of metrics - this will error.\n\n**CRITICAL - Local vs Uploaded Differences:**\n\n| | Local `evaluate()` | Uploaded to LangSmith |\n|---|---|---|\n| **Column name** | Python: auto-derived from function name. TypeScript: must include `key` field or column is untitled | Comes from evaluator name set at upload time. Do NOT include `key` — it creates a duplicate column |\n| **Python `run` type** | `RunTree` object → `run.outputs` (attribute) | `dict` → `run[\"outputs\"]` (subscript). Handle both: `run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {})` |\n| **TypeScript `run` type** | Always attribute access: `run.outputs?.field` | Always attribute access: `run.outputs?.field` |\n| **Python return** | `{\"score\": value, \"comment\": \"...\"}` | `{\"score\": value, \"comment\": \"...\"}` |\n| **TypeScript return** | `{ key: \"name\", score: value, comment: \"...\" }` | `{ score: value, comment: \"...\" }` |\n\u003C\u002Fevaluator_format>\n\n\u003Cevaluator_types>\n- **LLM as Judge** - Uses an LLM to grade outputs. Best for subjective quality (accuracy, helpfulness, relevance).\n- **Custom Code** - Deterministic logic. Best for objective checks (exact match, trajectory validation, format compliance).\n\u003C\u002Fevaluator_types>\n\n\u003Cllm_judge>\n## LLM as Judge Evaluators\n\n**NOTE:** LLM-as-Judge upload is currently not supported by the CLI — only code evaluators are supported. For evaluations against a dataset, STRONGLY PREFER defining local evaluators to use with `evaluate(evaluators=[...])`.\n\n\u003Cpython>\n```python\nfrom typing import TypedDict, Annotated\nfrom langchain_openai import ChatOpenAI\n\nclass Grade(TypedDict):\n    reasoning: Annotated[str, ..., \"Explain your reasoning\"]\n    is_accurate: Annotated[bool, ..., \"True if response is accurate\"]\n\njudge = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0).with_structured_output(Grade, method=\"json_schema\", strict=True)\n\nasync def accuracy_evaluator(run, example):\n    run_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\n    example_outputs = example.outputs if hasattr(example, \"outputs\") else example.get(\"outputs\", {}) or {}\n    grade = await judge.ainvoke([{\"role\": \"user\", \"content\": f\"Expected: {example_outputs}\\nActual: {run_outputs}\\nIs this accurate?\"}])\n    return {\"score\": 1 if grade[\"is_accurate\"] else 0, \"comment\": grade[\"reasoning\"]}\n```\n\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nimport OpenAI from \"openai\";\n\nconst openai = new OpenAI();\n\nasync function accuracyEvaluator(run, example) {\n    const runOutputs = run.outputs ?? {};\n    const exampleOutputs = example.outputs ?? {};\n\n    const response = await openai.chat.completions.create({\n    model: \"gpt-4o-mini\",\n    temperature: 0,\n    response_format: { type: \"json_object\" },\n    messages: [\n        { role: \"system\", content: 'Respond with JSON: {\"is_accurate\": boolean, \"reasoning\": string}' },\n        { role: \"user\", content: `Expected: ${JSON.stringify(exampleOutputs)}\\nActual: ${JSON.stringify(runOutputs)}\\nIs this accurate?` }\n    ]\n    });\n\n    const grade = JSON.parse(response.choices[0].message.content);\n    return { score: grade.is_accurate ? 1 : 0, comment: grade.reasoning };\n}\n```\n\u003C\u002Ftypescript>\n\u003C\u002Fllm_judge>\n\n\u003Ccode_evaluators>\n## Custom Code Evaluators\n\n**Before writing an evaluator:**\n1. Inspect your dataset to understand expected field names (see Golden Rule above)\n2. Test your run function and verify its output structure matches the dataset schema\n3. Query LangSmith traces to debug any mismatches\n\n\u003Cpython>\n```python\ndef trajectory_evaluator(run, example):\n    run_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\n    example_outputs = example.outputs if hasattr(example, \"outputs\") else example.get(\"outputs\", {}) or {}\n    # IMPORTANT: Replace these placeholders with your actual field names\n    # 1. Query your LangSmith trace to see what fields exist in run outputs\n    # 2. Check your dataset schema for expected field names\n    # Note: Trajectory data may not appear in default output - verify against trace!\n    actual = run_outputs.get(\"YOUR_TRAJECTORY_FIELD\", [])\n    expected = example_outputs.get(\"YOUR_EXPECTED_FIELD\", [])\n    return {\"score\": 1 if actual == expected else 0, \"comment\": f\"Expected {expected}, got {actual}\"}\n```\n\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nfunction trajectoryEvaluator(run, example) {\n    const runOutputs = run.outputs ?? {};\n    const exampleOutputs = example.outputs ?? {};\n    \u002F\u002F IMPORTANT: Replace these placeholders with your actual field names\n    \u002F\u002F 1. Query your LangSmith trace to see what fields exist in run outputs\n    \u002F\u002F 2. Check your dataset schema for expected field names\n    const actual = runOutputs.YOUR_TRAJECTORY_FIELD ?? [];\n    const expected = exampleOutputs.YOUR_EXPECTED_FIELD ?? [];\n    const match = JSON.stringify(actual) === JSON.stringify(expected);\n    return { score: match ? 1 : 0, comment: `Expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}` };\n}\n```\n\u003C\u002Ftypescript>\n\u003C\u002Fcode_evaluators>\n\n\u003Crun_functions>\n## Defining Run Functions\n\nRun functions execute your agent and return outputs for evaluation.\n\n**CRITICAL - Test Your Run Function First:**\nBefore writing evaluators, you MUST test your run function and inspect the actual output structure. Output shapes vary by framework, agent type, and configuration.\n\n**Debugging workflow:**\n1. Run your agent once on sample input\n2. Query the trace to see the execution structure\n3. Print the raw output and verify against trace to output contains the right data\n4. Adjust the run function as needed\n4. Verify your output matches your dataset schema\n\n**Try your hardest to match your run function output to your dataset schema.** This makes evaluators simple and reusable. If matching isn't possible, your evaluator must know how to extract and compare the right fields from each side.\n\n\u003Cpython>\n```python\ndef run_agent(inputs: dict) -> dict:\n    result = your_agent.run(inputs)\n    # ALWAYS inspect output shape first - run this, check the print, query traces\n    print(f\"DEBUG - type: {type(result)}, keys: {result.keys() if hasattr(result, 'keys') else 'N\u002FA'}\")\n    print(f\"DEBUG - value: {result}\")\n    return {\"output\": result}  # Adjust to match your dataset schema\n```\n\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nasync function runAgent(inputs) {\n    const result = await yourAgent.invoke(inputs);\n    \u002F\u002F ALWAYS inspect output shape first\n    console.log(\"DEBUG - type:\", typeof result, \"keys:\", Object.keys(result));\n    console.log(\"DEBUG - value:\", result);\n    return { output: result };  \u002F\u002F Adjust to match your dataset schema\n}\n```\n\u003C\u002Ftypescript>\n\n### Capturing Trajectories\n\nFor trajectory evaluation, your run function must capture tool calls during execution.\n\n**CRITICAL:** Run output formats vary significantly by framework and agent type. You MUST inspect before implementing:\n\n**LangGraph agents (LangChain OSS):** Use `stream_mode=\"debug\"` with `subgraphs=True` to capture nested subagent tool calls.\n\n```python\nimport uuid\n\ndef run_agent_with_trajectory(agent, inputs: dict) -> dict:\n    config = {\"configurable\": {\"thread_id\": f\"eval-{uuid.uuid4()}\"}}\n    trajectory = []\n    final_result = None\n\n    for chunk in agent.stream(inputs, config=config, stream_mode=\"debug\", subgraphs=True):\n        # STEP 1: Print chunks to understand the structure\n        print(f\"DEBUG chunk: {chunk}\")\n\n        # STEP 2: Write extraction based on YOUR observed structure\n        # ... your extraction logic here ...\n\n    # IMPORTANT: After running, query the LangSmith trace to verify\n    # your trajectory data is complete. Default output may be missing\n    # tool calls that appear in the trace.\n    return {\"output\": final_result, \"trajectory\": trajectory}\n```\n\n**Custom \u002F Non-LangChain Agents:**\n\n1. **Inspect output first** - Run your agent and inspect the result structure. Trajectory data may already be included in the output (e.g., `result.tool_calls`, `result.steps`, etc.)\n2. **Callbacks\u002FHooks** - If your framework supports execution callbacks, register a hook that records tool names on each invocation\n3. **Parse execution logs** - As a last resort, extract tool names from structured logs or trace data\n\nThe key is to capture the tool name at execution time, not at definition time.\n\u003C\u002Frun_functions>\n\n\u003Cupload>\n## Uploading Evaluators to LangSmith\n\n**IMPORTANT - Auto-Run Behavior:**\nEvaluators uploaded to a dataset **automatically run** when you run experiments on that dataset. You do NOT need to pass them to `evaluate()` - just run your agent against the dataset and the uploaded evaluators execute automatically.\n\n**IMPORTANT - Local vs Uploaded:**\nUploaded evaluators run in a sandboxed environment with very limited package access. Only use built-in\u002Fstandard library imports, and place all imports **inside** the evaluator function body. For dataset (offline) evaluators, prefer running locally with `evaluate(evaluators=[...])` first — this gives you full package access.\n\n**IMPORTANT - Code vs Structured Evaluators:**\n- **Code evaluators** (what the CLI uploads): Run in a limited environment without external packages. Use for deterministic logic (exact match, trajectory validation).\n- **Structured evaluators** (LLM-as-Judge): Configured via LangSmith UI, use a specific payload format with model\u002Fprompt\u002Fschema. The CLI does not support this format yet.\n\n**IMPORTANT - Choose the right target:**\n- `--dataset`: Offline evaluator with `(run, example)` signature - for comparing to expected values\n- `--project`: Online evaluator with `(run)` signature - for real-time quality checks\n\nYou must specify one. Global evaluators are not supported.\n\n```bash\n# List all evaluators\nlangsmith evaluator list --api-key $LANGSMITH_API_KEY\n\n# Upload offline evaluator (attached to dataset)\nlangsmith evaluator upload my_evaluators.py \\\n  --name \"Trajectory Match\" --function trajectory_evaluator \\\n  --dataset \"My Dataset\" --replace --api-key $LANGSMITH_API_KEY\n\n# Upload online evaluator (attached to project)\nlangsmith evaluator upload my_evaluators.py \\\n  --name \"Quality Check\" --function quality_check \\\n  --project \"Production Agent\" --replace --api-key $LANGSMITH_API_KEY\n\n# Delete\nlangsmith evaluator delete \"Trajectory Match\" --api-key $LANGSMITH_API_KEY\n```\n\n**IMPORTANT - Safety Prompts:**\n- The CLI prompts for confirmation before destructive operations\n- **NEVER use `--yes` flag unless the user explicitly requests it**\n\u003C\u002Fupload>\n\n\u003Cbest_practices>\n1. **Use structured output for LLM judges** - More reliable than parsing free-text\n2. **Match evaluator to dataset type**\n   - Final Response → LLM as Judge for quality\n   - Trajectory → Custom Code for sequence\n3. **Use async for LLM judges** - Enables parallel evaluation\n4. **Test evaluators independently** - Validate on known good\u002Fbad examples first\n5. **Choose the right language**\n   - Python: Use for Python agents, langchain integrations\n   - JavaScript: Use for TypeScript\u002FNode.js agents\n\u003C\u002Fbest_practices>\n\n\u003Crunning_evaluations>\n## Running Evaluations\n\n**Uploaded evaluators** auto-run when you run experiments - no code needed. **Local evaluators** are passed directly for development\u002Ftesting.\n\n\u003Cpython>\n```python\nfrom langsmith import evaluate\n\n# Uploaded evaluators run automatically\nresults = evaluate(run_agent, data=\"My Dataset\", experiment_prefix=\"eval-v1\")\n\n# Or pass local evaluators for testing\nresults = evaluate(run_agent, data=\"My Dataset\", evaluators=[my_evaluator], experiment_prefix=\"eval-v1\")\n```\n\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nimport { evaluate } from \"langsmith\u002Fevaluation\";\n\n\u002F\u002F Uploaded evaluators run automatically\nconst results = await evaluate(runAgent, {\n  data: \"My Dataset\",\n  experimentPrefix: \"eval-v1\",\n});\n\n\u002F\u002F Or pass local evaluators for testing\nconst results = await evaluate(runAgent, {\n  data: \"My Dataset\",\n  evaluators: [myEvaluator],\n  experimentPrefix: \"eval-v1\",\n});\n```\n\u003C\u002Ftypescript>\n\u003C\u002Frunning_evaluations>\n\n\u003Ctroubleshooting>\n## Common Issues\n\n**Output doesn't match what you expect:** Query the LangSmith trace. It shows exact inputs\u002Foutputs at each step - compare what you find to what you're trying to extract.\n\n**One metric per evaluator:** Return `{\"score\": value, \"comment\": \"...\"}`. For multiple metrics, create separate functions.\n\n**Field name mismatch:** Your run function output must match dataset schema exactly. Inspect dataset first with `client.read_example(example_id)`.\n\n**RunTree vs dict (Python only):** Local `evaluate()` passes `RunTree`, uploaded evaluators receive `dict`. Handle both:\n```python\nrun_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\n```\nTypeScript always uses attribute access: `run.outputs?.field`\n\u003C\u002Ftroubleshooting>\n\n\u003Cresources>\n- [LangSmith Evaluation Concepts](https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fevaluation-concepts)\n- [Custom Code Evaluators](https:\u002F\u002Fchangelog.langchain.com\u002Fannouncements\u002Fcustom-code-evaluators-in-langsmith)\n- [OpenEvals - Readymade Evaluators](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals)\n\u003C\u002Fresources>\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,46,358,363,370,380,415,420,425,431,441,474,484,514,522,550,558,802,807,830,835,841,859,2062],{"type":39,"tag":40,"props":41,"children":42},"element","oneliner",{},[43],{"type":44,"value":45},"text","\nThree core components: **(1) Creating Evaluators** - LLM-as-Judge, custom code; **(2) Defining Run Functions** - capture agent outputs\u002Ftrajectories for evaluation; **(3) Running Evaluations** - locally with `evaluate()` or auto-run via uploaded evaluators. Python and TypeScript examples included.\n",{"type":39,"tag":47,"props":48,"children":49},"setup",{},[50,52,163,184,219,245,250,285,290,325,330],{"type":44,"value":51},"\nEnvironment Variables\n",{"type":39,"tag":53,"props":54,"children":59},"pre",{"className":55,"code":56,"language":57,"meta":58,"style":58},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED\nLANGSMITH_PROJECT=your-project-name                   # Check this to know which project has traces\nLANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys\nOPENAI_API_KEY=your_openai_key                        # For LLM as Judge\n","bash","",[60],{"type":39,"tag":61,"props":62,"children":63},"code",{"__ignoreMap":58},[64,94,117,140],{"type":39,"tag":65,"props":66,"children":69},"span",{"class":67,"line":68},"line",1,[70,76,82,88],{"type":39,"tag":65,"props":71,"children":73},{"style":72},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[74],{"type":44,"value":75},"LANGSMITH_API_KEY",{"type":39,"tag":65,"props":77,"children":79},{"style":78},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[80],{"type":44,"value":81},"=",{"type":39,"tag":65,"props":83,"children":85},{"style":84},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[86],{"type":44,"value":87},"lsv2_pt_your_api_key_here",{"type":39,"tag":65,"props":89,"children":91},{"style":90},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[92],{"type":44,"value":93},"          # REQUIRED\n",{"type":39,"tag":65,"props":95,"children":97},{"class":67,"line":96},2,[98,103,107,112],{"type":39,"tag":65,"props":99,"children":100},{"style":72},[101],{"type":44,"value":102},"LANGSMITH_PROJECT",{"type":39,"tag":65,"props":104,"children":105},{"style":78},[106],{"type":44,"value":81},{"type":39,"tag":65,"props":108,"children":109},{"style":84},[110],{"type":44,"value":111},"your-project-name",{"type":39,"tag":65,"props":113,"children":114},{"style":90},[115],{"type":44,"value":116},"                   # Check this to know which project has traces\n",{"type":39,"tag":65,"props":118,"children":120},{"class":67,"line":119},3,[121,126,130,135],{"type":39,"tag":65,"props":122,"children":123},{"style":72},[124],{"type":44,"value":125},"LANGSMITH_WORKSPACE_ID",{"type":39,"tag":65,"props":127,"children":128},{"style":78},[129],{"type":44,"value":81},{"type":39,"tag":65,"props":131,"children":132},{"style":84},[133],{"type":44,"value":134},"your-workspace-id",{"type":39,"tag":65,"props":136,"children":137},{"style":90},[138],{"type":44,"value":139},"              # Optional: for org-scoped keys\n",{"type":39,"tag":65,"props":141,"children":143},{"class":67,"line":142},4,[144,149,153,158],{"type":39,"tag":65,"props":145,"children":146},{"style":72},[147],{"type":44,"value":148},"OPENAI_API_KEY",{"type":39,"tag":65,"props":150,"children":151},{"style":78},[152],{"type":44,"value":81},{"type":39,"tag":65,"props":154,"children":155},{"style":84},[156],{"type":44,"value":157},"your_openai_key",{"type":39,"tag":65,"props":159,"children":160},{"style":90},[161],{"type":44,"value":162},"                        # For LLM as Judge\n",{"type":39,"tag":164,"props":165,"children":166},"p",{},[167,169,174,176,182],{"type":44,"value":168},"Authentication is REQUIRED: either set the ",{"type":39,"tag":61,"props":170,"children":172},{"className":171},[],[173],{"type":44,"value":75},{"type":44,"value":175}," environment variable, or pass the ",{"type":39,"tag":61,"props":177,"children":179},{"className":178},[],[180],{"type":44,"value":181},"--api-key",{"type":44,"value":183}," flag to CLI commands (preferred):",{"type":39,"tag":53,"props":185,"children":187},{"className":55,"code":186,"language":57,"meta":58,"style":58},"langsmith evaluator list --api-key $LANGSMITH_API_KEY\n",[188],{"type":39,"tag":61,"props":189,"children":190},{"__ignoreMap":58},[191],{"type":39,"tag":65,"props":192,"children":193},{"class":67,"line":68},[194,199,204,209,214],{"type":39,"tag":65,"props":195,"children":197},{"style":196},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[198],{"type":44,"value":22},{"type":39,"tag":65,"props":200,"children":201},{"style":84},[202],{"type":44,"value":203}," evaluator",{"type":39,"tag":65,"props":205,"children":206},{"style":84},[207],{"type":44,"value":208}," list",{"type":39,"tag":65,"props":210,"children":211},{"style":84},[212],{"type":44,"value":213}," --api-key",{"type":39,"tag":65,"props":215,"children":216},{"style":72},[217],{"type":44,"value":218}," $LANGSMITH_API_KEY\n",{"type":39,"tag":164,"props":220,"children":221},{},[222,228,230,236,238,243],{"type":39,"tag":223,"props":224,"children":225},"strong",{},[226],{"type":44,"value":227},"IMPORTANT:",{"type":44,"value":229}," Always check the environment variables or ",{"type":39,"tag":61,"props":231,"children":233},{"className":232},[],[234],{"type":44,"value":235},".env",{"type":44,"value":237}," file for ",{"type":39,"tag":61,"props":239,"children":241},{"className":240},[],[242],{"type":44,"value":102},{"type":44,"value":244}," before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.",{"type":39,"tag":164,"props":246,"children":247},{},[248],{"type":44,"value":249},"Python Dependencies",{"type":39,"tag":53,"props":251,"children":253},{"className":55,"code":252,"language":57,"meta":58,"style":58},"pip install langsmith langchain-openai python-dotenv\n",[254],{"type":39,"tag":61,"props":255,"children":256},{"__ignoreMap":58},[257],{"type":39,"tag":65,"props":258,"children":259},{"class":67,"line":68},[260,265,270,275,280],{"type":39,"tag":65,"props":261,"children":262},{"style":196},[263],{"type":44,"value":264},"pip",{"type":39,"tag":65,"props":266,"children":267},{"style":84},[268],{"type":44,"value":269}," install",{"type":39,"tag":65,"props":271,"children":272},{"style":84},[273],{"type":44,"value":274}," langsmith",{"type":39,"tag":65,"props":276,"children":277},{"style":84},[278],{"type":44,"value":279}," langchain-openai",{"type":39,"tag":65,"props":281,"children":282},{"style":84},[283],{"type":44,"value":284}," python-dotenv\n",{"type":39,"tag":164,"props":286,"children":287},{},[288],{"type":44,"value":289},"CLI Tool (for uploading evaluators)",{"type":39,"tag":53,"props":291,"children":293},{"className":55,"code":292,"language":57,"meta":58,"style":58},"curl -sSL https:\u002F\u002Fraw.githubusercontent.com\u002Flangchain-ai\u002Flangsmith-cli\u002Fmain\u002Fscripts\u002Finstall.sh | sh\n",[294],{"type":39,"tag":61,"props":295,"children":296},{"__ignoreMap":58},[297],{"type":39,"tag":65,"props":298,"children":299},{"class":67,"line":68},[300,305,310,315,320],{"type":39,"tag":65,"props":301,"children":302},{"style":196},[303],{"type":44,"value":304},"curl",{"type":39,"tag":65,"props":306,"children":307},{"style":84},[308],{"type":44,"value":309}," -sSL",{"type":39,"tag":65,"props":311,"children":312},{"style":84},[313],{"type":44,"value":314}," https:\u002F\u002Fraw.githubusercontent.com\u002Flangchain-ai\u002Flangsmith-cli\u002Fmain\u002Fscripts\u002Finstall.sh",{"type":39,"tag":65,"props":316,"children":317},{"style":78},[318],{"type":44,"value":319}," |",{"type":39,"tag":65,"props":321,"children":322},{"style":196},[323],{"type":44,"value":324}," sh\n",{"type":39,"tag":164,"props":326,"children":327},{},[328],{"type":44,"value":329},"JavaScript Dependencies",{"type":39,"tag":53,"props":331,"children":333},{"className":55,"code":332,"language":57,"meta":58,"style":58},"npm install langsmith openai\n",[334],{"type":39,"tag":61,"props":335,"children":336},{"__ignoreMap":58},[337],{"type":39,"tag":65,"props":338,"children":339},{"class":67,"line":68},[340,345,349,353],{"type":39,"tag":65,"props":341,"children":342},{"style":196},[343],{"type":44,"value":344},"npm",{"type":39,"tag":65,"props":346,"children":347},{"style":84},[348],{"type":44,"value":269},{"type":39,"tag":65,"props":350,"children":351},{"style":84},[352],{"type":44,"value":274},{"type":39,"tag":65,"props":354,"children":355},{"style":84},[356],{"type":44,"value":357}," openai\n",{"type":39,"tag":164,"props":359,"children":360},{},[361],{"type":44,"value":362},"\u003Ccrucial_requirement>",{"type":39,"tag":364,"props":365,"children":367},"h2",{"id":366},"golden-rule-inspect-before-you-implement",[368],{"type":44,"value":369},"Golden Rule: Inspect Before You Implement",{"type":39,"tag":164,"props":371,"children":372},{},[373,378],{"type":39,"tag":223,"props":374,"children":375},{},[376],{"type":44,"value":377},"CRITICAL:",{"type":44,"value":379}," Before writing ANY evaluator or extraction logic, you MUST:",{"type":39,"tag":381,"props":382,"children":383},"ol",{},[384,395,405],{"type":39,"tag":385,"props":386,"children":387},"li",{},[388,393],{"type":39,"tag":223,"props":389,"children":390},{},[391],{"type":44,"value":392},"Run your agent",{"type":44,"value":394}," on sample inputs and capture the actual output",{"type":39,"tag":385,"props":396,"children":397},{},[398,403],{"type":39,"tag":223,"props":399,"children":400},{},[401],{"type":44,"value":402},"Inspect the output",{"type":44,"value":404}," - print it, query LangSmith traces, understand the exact structure",{"type":39,"tag":385,"props":406,"children":407},{},[408,413],{"type":39,"tag":223,"props":409,"children":410},{},[411],{"type":44,"value":412},"Only then",{"type":44,"value":414}," write code that processes that output",{"type":39,"tag":164,"props":416,"children":417},{},[418],{"type":44,"value":419},"Output structures vary significantly by framework, agent type, and configuration. Never assume the shape - always verify first. Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution.\n\u003C\u002Fcrucial_requirement>",{"type":39,"tag":164,"props":421,"children":422},{},[423],{"type":44,"value":424},"\u003Cevaluator_format>",{"type":39,"tag":364,"props":426,"children":428},{"id":427},"offline-vs-online-evaluators",[429],{"type":44,"value":430},"Offline vs Online Evaluators",{"type":39,"tag":164,"props":432,"children":433},{},[434,439],{"type":39,"tag":223,"props":435,"children":436},{},[437],{"type":44,"value":438},"Offline Evaluators",{"type":44,"value":440}," (attached to datasets):",{"type":39,"tag":442,"props":443,"children":444},"ul",{},[445,458,463],{"type":39,"tag":385,"props":446,"children":447},{},[448,450,456],{"type":44,"value":449},"Function signature: ",{"type":39,"tag":61,"props":451,"children":453},{"className":452},[],[454],{"type":44,"value":455},"(run, example)",{"type":44,"value":457}," - receives both run outputs and dataset example",{"type":39,"tag":385,"props":459,"children":460},{},[461],{"type":44,"value":462},"Use case: Comparing agent outputs to expected values in a dataset",{"type":39,"tag":385,"props":464,"children":465},{},[466,468],{"type":44,"value":467},"Upload with: ",{"type":39,"tag":61,"props":469,"children":471},{"className":470},[],[472],{"type":44,"value":473},"--dataset \"Dataset Name\"",{"type":39,"tag":164,"props":475,"children":476},{},[477,482],{"type":39,"tag":223,"props":478,"children":479},{},[480],{"type":44,"value":481},"Online Evaluators",{"type":44,"value":483}," (attached to projects):",{"type":39,"tag":442,"props":485,"children":486},{},[487,499,504],{"type":39,"tag":385,"props":488,"children":489},{},[490,491,497],{"type":44,"value":449},{"type":39,"tag":61,"props":492,"children":494},{"className":493},[],[495],{"type":44,"value":496},"(run)",{"type":44,"value":498}," - receives only run outputs, NO example parameter",{"type":39,"tag":385,"props":500,"children":501},{},[502],{"type":44,"value":503},"Use case: Real-time quality checks on production runs (no reference data)",{"type":39,"tag":385,"props":505,"children":506},{},[507,508],{"type":44,"value":467},{"type":39,"tag":61,"props":509,"children":511},{"className":510},[],[512],{"type":44,"value":513},"--project \"Project Name\"",{"type":39,"tag":164,"props":515,"children":516},{},[517],{"type":39,"tag":223,"props":518,"children":519},{},[520],{"type":44,"value":521},"CRITICAL - Return Format:",{"type":39,"tag":442,"props":523,"children":524},{},[525,537],{"type":39,"tag":385,"props":526,"children":527},{},[528,530,535],{"type":44,"value":529},"Each evaluator returns ",{"type":39,"tag":223,"props":531,"children":532},{},[533],{"type":44,"value":534},"ONE metric only",{"type":44,"value":536},". For multiple metrics, create multiple evaluator functions.",{"type":39,"tag":385,"props":538,"children":539},{},[540,542,548],{"type":44,"value":541},"Do NOT return ",{"type":39,"tag":61,"props":543,"children":545},{"className":544},[],[546],{"type":44,"value":547},"{\"metric_name\": value}",{"type":44,"value":549}," or lists of metrics - this will error.",{"type":39,"tag":164,"props":551,"children":552},{},[553],{"type":39,"tag":223,"props":554,"children":555},{},[556],{"type":44,"value":557},"CRITICAL - Local vs Uploaded Differences:",{"type":39,"tag":559,"props":560,"children":561},"table",{},[562,590],{"type":39,"tag":563,"props":564,"children":565},"thead",{},[566],{"type":39,"tag":567,"props":568,"children":569},"tr",{},[570,574,585],{"type":39,"tag":571,"props":572,"children":573},"th",{},[],{"type":39,"tag":571,"props":575,"children":576},{},[577,579],{"type":44,"value":578},"Local ",{"type":39,"tag":61,"props":580,"children":582},{"className":581},[],[583],{"type":44,"value":584},"evaluate()",{"type":39,"tag":571,"props":586,"children":587},{},[588],{"type":44,"value":589},"Uploaded to LangSmith",{"type":39,"tag":591,"props":592,"children":593},"tbody",{},[594,631,694,731,759,788],{"type":39,"tag":567,"props":595,"children":596},{},[597,606,619],{"type":39,"tag":598,"props":599,"children":600},"td",{},[601],{"type":39,"tag":223,"props":602,"children":603},{},[604],{"type":44,"value":605},"Column name",{"type":39,"tag":598,"props":607,"children":608},{},[609,611,617],{"type":44,"value":610},"Python: auto-derived from function name. TypeScript: must include ",{"type":39,"tag":61,"props":612,"children":614},{"className":613},[],[615],{"type":44,"value":616},"key",{"type":44,"value":618}," field or column is untitled",{"type":39,"tag":598,"props":620,"children":621},{},[622,624,629],{"type":44,"value":623},"Comes from evaluator name set at upload time. Do NOT include ",{"type":39,"tag":61,"props":625,"children":627},{"className":626},[],[628],{"type":44,"value":616},{"type":44,"value":630}," — it creates a duplicate column",{"type":39,"tag":567,"props":632,"children":633},{},[634,650,669],{"type":39,"tag":598,"props":635,"children":636},{},[637],{"type":39,"tag":223,"props":638,"children":639},{},[640,642,648],{"type":44,"value":641},"Python ",{"type":39,"tag":61,"props":643,"children":645},{"className":644},[],[646],{"type":44,"value":647},"run",{"type":44,"value":649}," type",{"type":39,"tag":598,"props":651,"children":652},{},[653,659,661,667],{"type":39,"tag":61,"props":654,"children":656},{"className":655},[],[657],{"type":44,"value":658},"RunTree",{"type":44,"value":660}," object → ",{"type":39,"tag":61,"props":662,"children":664},{"className":663},[],[665],{"type":44,"value":666},"run.outputs",{"type":44,"value":668}," (attribute)",{"type":39,"tag":598,"props":670,"children":671},{},[672,678,680,686,688],{"type":39,"tag":61,"props":673,"children":675},{"className":674},[],[676],{"type":44,"value":677},"dict",{"type":44,"value":679}," → ",{"type":39,"tag":61,"props":681,"children":683},{"className":682},[],[684],{"type":44,"value":685},"run[\"outputs\"]",{"type":44,"value":687}," (subscript). Handle both: ",{"type":39,"tag":61,"props":689,"children":691},{"className":690},[],[692],{"type":44,"value":693},"run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {})",{"type":39,"tag":567,"props":695,"children":696},{},[697,711,722],{"type":39,"tag":598,"props":698,"children":699},{},[700],{"type":39,"tag":223,"props":701,"children":702},{},[703,705,710],{"type":44,"value":704},"TypeScript ",{"type":39,"tag":61,"props":706,"children":708},{"className":707},[],[709],{"type":44,"value":647},{"type":44,"value":649},{"type":39,"tag":598,"props":712,"children":713},{},[714,716],{"type":44,"value":715},"Always attribute access: ",{"type":39,"tag":61,"props":717,"children":719},{"className":718},[],[720],{"type":44,"value":721},"run.outputs?.field",{"type":39,"tag":598,"props":723,"children":724},{},[725,726],{"type":44,"value":715},{"type":39,"tag":61,"props":727,"children":729},{"className":728},[],[730],{"type":44,"value":721},{"type":39,"tag":567,"props":732,"children":733},{},[734,742,751],{"type":39,"tag":598,"props":735,"children":736},{},[737],{"type":39,"tag":223,"props":738,"children":739},{},[740],{"type":44,"value":741},"Python return",{"type":39,"tag":598,"props":743,"children":744},{},[745],{"type":39,"tag":61,"props":746,"children":748},{"className":747},[],[749],{"type":44,"value":750},"{\"score\": value, \"comment\": \"...\"}",{"type":39,"tag":598,"props":752,"children":753},{},[754],{"type":39,"tag":61,"props":755,"children":757},{"className":756},[],[758],{"type":44,"value":750},{"type":39,"tag":567,"props":760,"children":761},{},[762,770,779],{"type":39,"tag":598,"props":763,"children":764},{},[765],{"type":39,"tag":223,"props":766,"children":767},{},[768],{"type":44,"value":769},"TypeScript return",{"type":39,"tag":598,"props":771,"children":772},{},[773],{"type":39,"tag":61,"props":774,"children":776},{"className":775},[],[777],{"type":44,"value":778},"{ key: \"name\", score: value, comment: \"...\" }",{"type":39,"tag":598,"props":780,"children":781},{},[782],{"type":39,"tag":61,"props":783,"children":785},{"className":784},[],[786],{"type":44,"value":787},"{ score: value, comment: \"...\" }",{"type":39,"tag":567,"props":789,"children":790},{},[791,796,799],{"type":39,"tag":598,"props":792,"children":793},{},[794],{"type":44,"value":795},"\u003C\u002Fevaluator_format>",{"type":39,"tag":598,"props":797,"children":798},{},[],{"type":39,"tag":598,"props":800,"children":801},{},[],{"type":39,"tag":164,"props":803,"children":804},{},[805],{"type":44,"value":806},"\u003Cevaluator_types>",{"type":39,"tag":442,"props":808,"children":809},{},[810,820],{"type":39,"tag":385,"props":811,"children":812},{},[813,818],{"type":39,"tag":223,"props":814,"children":815},{},[816],{"type":44,"value":817},"LLM as Judge",{"type":44,"value":819}," - Uses an LLM to grade outputs. Best for subjective quality (accuracy, helpfulness, relevance).",{"type":39,"tag":385,"props":821,"children":822},{},[823,828],{"type":39,"tag":223,"props":824,"children":825},{},[826],{"type":44,"value":827},"Custom Code",{"type":44,"value":829}," - Deterministic logic. Best for objective checks (exact match, trajectory validation, format compliance).\n\u003C\u002Fevaluator_types>",{"type":39,"tag":164,"props":831,"children":832},{},[833],{"type":44,"value":834},"\u003Cllm_judge>",{"type":39,"tag":364,"props":836,"children":838},{"id":837},"llm-as-judge-evaluators",[839],{"type":44,"value":840},"LLM as Judge Evaluators",{"type":39,"tag":164,"props":842,"children":843},{},[844,849,851,857],{"type":39,"tag":223,"props":845,"children":846},{},[847],{"type":44,"value":848},"NOTE:",{"type":44,"value":850}," LLM-as-Judge upload is currently not supported by the CLI — only code evaluators are supported. For evaluations against a dataset, STRONGLY PREFER defining local evaluators to use with ",{"type":39,"tag":61,"props":852,"children":854},{"className":853},[],[855],{"type":44,"value":856},"evaluate(evaluators=[...])",{"type":44,"value":858},".",{"type":39,"tag":860,"props":861,"children":862},"python",{},[863,865,882,887,913,923,928,934,942,960,965,971,976,982,987,997,1005,1033,1043,1048,1053,1060,1065,1074,1100,1263,1271,1320,1325],{"type":44,"value":864},"\n```python\nfrom typing import TypedDict, Annotated\nfrom langchain_openai import ChatOpenAI\n",{"type":39,"tag":164,"props":866,"children":867},{},[868,870,875,877],{"type":44,"value":869},"class Grade(TypedDict):\nreasoning: Annotated",{"type":39,"tag":65,"props":871,"children":872},{},[873],{"type":44,"value":874},"str, ..., \"Explain your reasoning\"",{"type":44,"value":876},"\nis_accurate: Annotated",{"type":39,"tag":65,"props":878,"children":879},{},[880],{"type":44,"value":881},"bool, ..., \"True if response is accurate\"",{"type":39,"tag":164,"props":883,"children":884},{},[885],{"type":44,"value":886},"judge = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0).with_structured_output(Grade, method=\"json_schema\", strict=True)",{"type":39,"tag":164,"props":888,"children":889},{},[890,892,897,899,904,906,911],{"type":44,"value":891},"async def accuracy_evaluator(run, example):\nrun_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\nexample_outputs = example.outputs if hasattr(example, \"outputs\") else example.get(\"outputs\", {}) or {}\ngrade = await judge.ainvoke(",{"type":39,"tag":65,"props":893,"children":894},{},[895],{"type":44,"value":896},"{\"role\": \"user\", \"content\": f\"Expected: {example_outputs}\\nActual: {run_outputs}\\nIs this accurate?\"}",{"type":44,"value":898},")\nreturn {\"score\": 1 if grade",{"type":39,"tag":65,"props":900,"children":901},{},[902],{"type":44,"value":903},"\"is_accurate\"",{"type":44,"value":905}," else 0, \"comment\": grade",{"type":39,"tag":65,"props":907,"children":908},{},[909],{"type":44,"value":910},"\"reasoning\"",{"type":44,"value":912},"}",{"type":39,"tag":53,"props":914,"children":918},{"className":915,"code":917,"language":44},[916],"language-text","\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nimport OpenAI from \"openai\";\n\nconst openai = new OpenAI();\n\nasync function accuracyEvaluator(run, example) {\n    const runOutputs = run.outputs ?? {};\n    const exampleOutputs = example.outputs ?? {};\n\n    const response = await openai.chat.completions.create({\n    model: \"gpt-4o-mini\",\n    temperature: 0,\n    response_format: { type: \"json_object\" },\n    messages: [\n        { role: \"system\", content: 'Respond with JSON: {\"is_accurate\": boolean, \"reasoning\": string}' },\n        { role: \"user\", content: `Expected: ${JSON.stringify(exampleOutputs)}\\nActual: ${JSON.stringify(runOutputs)}\\nIs this accurate?` }\n    ]\n    });\n\n    const grade = JSON.parse(response.choices[0].message.content);\n    return { score: grade.is_accurate ? 1 : 0, comment: grade.reasoning };\n}\n",[919],{"type":39,"tag":61,"props":920,"children":921},{"__ignoreMap":58},[922],{"type":44,"value":917},{"type":39,"tag":164,"props":924,"children":925},{},[926],{"type":44,"value":927},"\u003Ccode_evaluators>",{"type":39,"tag":364,"props":929,"children":931},{"id":930},"custom-code-evaluators",[932],{"type":44,"value":933},"Custom Code Evaluators",{"type":39,"tag":164,"props":935,"children":936},{},[937],{"type":39,"tag":223,"props":938,"children":939},{},[940],{"type":44,"value":941},"Before writing an evaluator:",{"type":39,"tag":381,"props":943,"children":944},{},[945,950,955],{"type":39,"tag":385,"props":946,"children":947},{},[948],{"type":44,"value":949},"Inspect your dataset to understand expected field names (see Golden Rule above)",{"type":39,"tag":385,"props":951,"children":952},{},[953],{"type":44,"value":954},"Test your run function and verify its output structure matches the dataset schema",{"type":39,"tag":385,"props":956,"children":957},{},[958],{"type":44,"value":959},"Query LangSmith traces to debug any mismatches",{"type":39,"tag":860,"props":961,"children":962},{},[963],{"type":44,"value":964},"\n```python\ndef trajectory_evaluator(run, example):\n    run_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\n    example_outputs = example.outputs if hasattr(example, \"outputs\") else example.get(\"outputs\", {}) or {}\n    # IMPORTANT: Replace these placeholders with your actual field names\n    # 1. Query your LangSmith trace to see what fields exist in run outputs\n    # 2. Check your dataset schema for expected field names\n    # Note: Trajectory data may not appear in default output - verify against trace!\n    actual = run_outputs.get(\"YOUR_TRAJECTORY_FIELD\", [])\n    expected = example_outputs.get(\"YOUR_EXPECTED_FIELD\", [])\n    return {\"score\": 1 if actual == expected else 0, \"comment\": f\"Expected {expected}, got {actual}\"}\n```\n",{"type":39,"tag":966,"props":967,"children":968},"typescript",{},[969],{"type":44,"value":970},"\n```javascript\nfunction trajectoryEvaluator(run, example) {\n    const runOutputs = run.outputs ?? {};\n    const exampleOutputs = example.outputs ?? {};\n    \u002F\u002F IMPORTANT: Replace these placeholders with your actual field names\n    \u002F\u002F 1. Query your LangSmith trace to see what fields exist in run outputs\n    \u002F\u002F 2. Check your dataset schema for expected field names\n    const actual = runOutputs.YOUR_TRAJECTORY_FIELD ?? [];\n    const expected = exampleOutputs.YOUR_EXPECTED_FIELD ?? [];\n    const match = JSON.stringify(actual) === JSON.stringify(expected);\n    return { score: match ? 1 : 0, comment: `Expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}` };\n}\n```\n",{"type":39,"tag":164,"props":972,"children":973},{},[974],{"type":44,"value":975},"\u003Crun_functions>",{"type":39,"tag":364,"props":977,"children":979},{"id":978},"defining-run-functions",[980],{"type":44,"value":981},"Defining Run Functions",{"type":39,"tag":164,"props":983,"children":984},{},[985],{"type":44,"value":986},"Run functions execute your agent and return outputs for evaluation.",{"type":39,"tag":164,"props":988,"children":989},{},[990,995],{"type":39,"tag":223,"props":991,"children":992},{},[993],{"type":44,"value":994},"CRITICAL - Test Your Run Function First:",{"type":44,"value":996},"\nBefore writing evaluators, you MUST test your run function and inspect the actual output structure. Output shapes vary by framework, agent type, and configuration.",{"type":39,"tag":164,"props":998,"children":999},{},[1000],{"type":39,"tag":223,"props":1001,"children":1002},{},[1003],{"type":44,"value":1004},"Debugging workflow:",{"type":39,"tag":381,"props":1006,"children":1007},{},[1008,1013,1018,1023,1028],{"type":39,"tag":385,"props":1009,"children":1010},{},[1011],{"type":44,"value":1012},"Run your agent once on sample input",{"type":39,"tag":385,"props":1014,"children":1015},{},[1016],{"type":44,"value":1017},"Query the trace to see the execution structure",{"type":39,"tag":385,"props":1019,"children":1020},{},[1021],{"type":44,"value":1022},"Print the raw output and verify against trace to output contains the right data",{"type":39,"tag":385,"props":1024,"children":1025},{},[1026],{"type":44,"value":1027},"Adjust the run function as needed",{"type":39,"tag":385,"props":1029,"children":1030},{},[1031],{"type":44,"value":1032},"Verify your output matches your dataset schema",{"type":39,"tag":164,"props":1034,"children":1035},{},[1036,1041],{"type":39,"tag":223,"props":1037,"children":1038},{},[1039],{"type":44,"value":1040},"Try your hardest to match your run function output to your dataset schema.",{"type":44,"value":1042}," This makes evaluators simple and reusable. If matching isn't possible, your evaluator must know how to extract and compare the right fields from each side.",{"type":39,"tag":860,"props":1044,"children":1045},{},[1046],{"type":44,"value":1047},"\n```python\ndef run_agent(inputs: dict) -> dict:\n    result = your_agent.run(inputs)\n    # ALWAYS inspect output shape first - run this, check the print, query traces\n    print(f\"DEBUG - type: {type(result)}, keys: {result.keys() if hasattr(result, 'keys') else 'N\u002FA'}\")\n    print(f\"DEBUG - value: {result}\")\n    return {\"output\": result}  # Adjust to match your dataset schema\n```\n",{"type":39,"tag":966,"props":1049,"children":1050},{},[1051],{"type":44,"value":1052},"\n```javascript\nasync function runAgent(inputs) {\n    const result = await yourAgent.invoke(inputs);\n    \u002F\u002F ALWAYS inspect output shape first\n    console.log(\"DEBUG - type:\", typeof result, \"keys:\", Object.keys(result));\n    console.log(\"DEBUG - value:\", result);\n    return { output: result };  \u002F\u002F Adjust to match your dataset schema\n}\n```\n",{"type":39,"tag":1054,"props":1055,"children":1057},"h3",{"id":1056},"capturing-trajectories",[1058],{"type":44,"value":1059},"Capturing Trajectories",{"type":39,"tag":164,"props":1061,"children":1062},{},[1063],{"type":44,"value":1064},"For trajectory evaluation, your run function must capture tool calls during execution.",{"type":39,"tag":164,"props":1066,"children":1067},{},[1068,1072],{"type":39,"tag":223,"props":1069,"children":1070},{},[1071],{"type":44,"value":377},{"type":44,"value":1073}," Run output formats vary significantly by framework and agent type. You MUST inspect before implementing:",{"type":39,"tag":164,"props":1075,"children":1076},{},[1077,1082,1084,1090,1092,1098],{"type":39,"tag":223,"props":1078,"children":1079},{},[1080],{"type":44,"value":1081},"LangGraph agents (LangChain OSS):",{"type":44,"value":1083}," Use ",{"type":39,"tag":61,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":44,"value":1089},"stream_mode=\"debug\"",{"type":44,"value":1091}," with ",{"type":39,"tag":61,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":44,"value":1097},"subgraphs=True",{"type":44,"value":1099}," to capture nested subagent tool calls.",{"type":39,"tag":53,"props":1101,"children":1104},{"className":1102,"code":1103,"language":860,"meta":58,"style":58},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import uuid\n\ndef run_agent_with_trajectory(agent, inputs: dict) -> dict:\n    config = {\"configurable\": {\"thread_id\": f\"eval-{uuid.uuid4()}\"}}\n    trajectory = []\n    final_result = None\n\n    for chunk in agent.stream(inputs, config=config, stream_mode=\"debug\", subgraphs=True):\n        # STEP 1: Print chunks to understand the structure\n        print(f\"DEBUG chunk: {chunk}\")\n\n        # STEP 2: Write extraction based on YOUR observed structure\n        # ... your extraction logic here ...\n\n    # IMPORTANT: After running, query the LangSmith trace to verify\n    # your trajectory data is complete. Default output may be missing\n    # tool calls that appear in the trace.\n    return {\"output\": final_result, \"trajectory\": trajectory}\n",[1105],{"type":39,"tag":61,"props":1106,"children":1107},{"__ignoreMap":58},[1108,1116,1125,1133,1141,1150,1159,1167,1176,1185,1194,1202,1211,1220,1228,1237,1246,1254],{"type":39,"tag":65,"props":1109,"children":1110},{"class":67,"line":68},[1111],{"type":39,"tag":65,"props":1112,"children":1113},{},[1114],{"type":44,"value":1115},"import uuid\n",{"type":39,"tag":65,"props":1117,"children":1118},{"class":67,"line":96},[1119],{"type":39,"tag":65,"props":1120,"children":1122},{"emptyLinePlaceholder":1121},true,[1123],{"type":44,"value":1124},"\n",{"type":39,"tag":65,"props":1126,"children":1127},{"class":67,"line":119},[1128],{"type":39,"tag":65,"props":1129,"children":1130},{},[1131],{"type":44,"value":1132},"def run_agent_with_trajectory(agent, inputs: dict) -> dict:\n",{"type":39,"tag":65,"props":1134,"children":1135},{"class":67,"line":142},[1136],{"type":39,"tag":65,"props":1137,"children":1138},{},[1139],{"type":44,"value":1140},"    config = {\"configurable\": {\"thread_id\": f\"eval-{uuid.uuid4()}\"}}\n",{"type":39,"tag":65,"props":1142,"children":1144},{"class":67,"line":1143},5,[1145],{"type":39,"tag":65,"props":1146,"children":1147},{},[1148],{"type":44,"value":1149},"    trajectory = []\n",{"type":39,"tag":65,"props":1151,"children":1153},{"class":67,"line":1152},6,[1154],{"type":39,"tag":65,"props":1155,"children":1156},{},[1157],{"type":44,"value":1158},"    final_result = None\n",{"type":39,"tag":65,"props":1160,"children":1162},{"class":67,"line":1161},7,[1163],{"type":39,"tag":65,"props":1164,"children":1165},{"emptyLinePlaceholder":1121},[1166],{"type":44,"value":1124},{"type":39,"tag":65,"props":1168,"children":1170},{"class":67,"line":1169},8,[1171],{"type":39,"tag":65,"props":1172,"children":1173},{},[1174],{"type":44,"value":1175},"    for chunk in agent.stream(inputs, config=config, stream_mode=\"debug\", subgraphs=True):\n",{"type":39,"tag":65,"props":1177,"children":1179},{"class":67,"line":1178},9,[1180],{"type":39,"tag":65,"props":1181,"children":1182},{},[1183],{"type":44,"value":1184},"        # STEP 1: Print chunks to understand the structure\n",{"type":39,"tag":65,"props":1186,"children":1188},{"class":67,"line":1187},10,[1189],{"type":39,"tag":65,"props":1190,"children":1191},{},[1192],{"type":44,"value":1193},"        print(f\"DEBUG chunk: {chunk}\")\n",{"type":39,"tag":65,"props":1195,"children":1197},{"class":67,"line":1196},11,[1198],{"type":39,"tag":65,"props":1199,"children":1200},{"emptyLinePlaceholder":1121},[1201],{"type":44,"value":1124},{"type":39,"tag":65,"props":1203,"children":1205},{"class":67,"line":1204},12,[1206],{"type":39,"tag":65,"props":1207,"children":1208},{},[1209],{"type":44,"value":1210},"        # STEP 2: Write extraction based on YOUR observed structure\n",{"type":39,"tag":65,"props":1212,"children":1214},{"class":67,"line":1213},13,[1215],{"type":39,"tag":65,"props":1216,"children":1217},{},[1218],{"type":44,"value":1219},"        # ... your extraction logic here ...\n",{"type":39,"tag":65,"props":1221,"children":1223},{"class":67,"line":1222},14,[1224],{"type":39,"tag":65,"props":1225,"children":1226},{"emptyLinePlaceholder":1121},[1227],{"type":44,"value":1124},{"type":39,"tag":65,"props":1229,"children":1231},{"class":67,"line":1230},15,[1232],{"type":39,"tag":65,"props":1233,"children":1234},{},[1235],{"type":44,"value":1236},"    # IMPORTANT: After running, query the LangSmith trace to verify\n",{"type":39,"tag":65,"props":1238,"children":1240},{"class":67,"line":1239},16,[1241],{"type":39,"tag":65,"props":1242,"children":1243},{},[1244],{"type":44,"value":1245},"    # your trajectory data is complete. Default output may be missing\n",{"type":39,"tag":65,"props":1247,"children":1248},{"class":67,"line":27},[1249],{"type":39,"tag":65,"props":1250,"children":1251},{},[1252],{"type":44,"value":1253},"    # tool calls that appear in the trace.\n",{"type":39,"tag":65,"props":1255,"children":1257},{"class":67,"line":1256},18,[1258],{"type":39,"tag":65,"props":1259,"children":1260},{},[1261],{"type":44,"value":1262},"    return {\"output\": final_result, \"trajectory\": trajectory}\n",{"type":39,"tag":164,"props":1264,"children":1265},{},[1266],{"type":39,"tag":223,"props":1267,"children":1268},{},[1269],{"type":44,"value":1270},"Custom \u002F Non-LangChain Agents:",{"type":39,"tag":381,"props":1272,"children":1273},{},[1274,1300,1310],{"type":39,"tag":385,"props":1275,"children":1276},{},[1277,1282,1284,1290,1292,1298],{"type":39,"tag":223,"props":1278,"children":1279},{},[1280],{"type":44,"value":1281},"Inspect output first",{"type":44,"value":1283}," - Run your agent and inspect the result structure. Trajectory data may already be included in the output (e.g., ",{"type":39,"tag":61,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":44,"value":1289},"result.tool_calls",{"type":44,"value":1291},", ",{"type":39,"tag":61,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":44,"value":1297},"result.steps",{"type":44,"value":1299},", etc.)",{"type":39,"tag":385,"props":1301,"children":1302},{},[1303,1308],{"type":39,"tag":223,"props":1304,"children":1305},{},[1306],{"type":44,"value":1307},"Callbacks\u002FHooks",{"type":44,"value":1309}," - If your framework supports execution callbacks, register a hook that records tool names on each invocation",{"type":39,"tag":385,"props":1311,"children":1312},{},[1313,1318],{"type":39,"tag":223,"props":1314,"children":1315},{},[1316],{"type":44,"value":1317},"Parse execution logs",{"type":44,"value":1319}," - As a last resort, extract tool names from structured logs or trace data",{"type":39,"tag":164,"props":1321,"children":1322},{},[1323],{"type":44,"value":1324},"The key is to capture the tool name at execution time, not at definition time.\n\u003C\u002Frun_functions>",{"type":39,"tag":1326,"props":1327,"children":1328},"upload",{},[1329,1331,1355,1379,1387,1410,1418,1457,1462,1767,1775,1799,1804,1879,1884,1890,1907],{"type":44,"value":1330},"\n## Uploading Evaluators to LangSmith\n",{"type":39,"tag":164,"props":1332,"children":1333},{},[1334,1339,1341,1346,1348,1353],{"type":39,"tag":223,"props":1335,"children":1336},{},[1337],{"type":44,"value":1338},"IMPORTANT - Auto-Run Behavior:",{"type":44,"value":1340},"\nEvaluators uploaded to a dataset ",{"type":39,"tag":223,"props":1342,"children":1343},{},[1344],{"type":44,"value":1345},"automatically run",{"type":44,"value":1347}," when you run experiments on that dataset. You do NOT need to pass them to ",{"type":39,"tag":61,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":44,"value":584},{"type":44,"value":1354}," - just run your agent against the dataset and the uploaded evaluators execute automatically.",{"type":39,"tag":164,"props":1356,"children":1357},{},[1358,1363,1365,1370,1372,1377],{"type":39,"tag":223,"props":1359,"children":1360},{},[1361],{"type":44,"value":1362},"IMPORTANT - Local vs Uploaded:",{"type":44,"value":1364},"\nUploaded evaluators run in a sandboxed environment with very limited package access. Only use built-in\u002Fstandard library imports, and place all imports ",{"type":39,"tag":223,"props":1366,"children":1367},{},[1368],{"type":44,"value":1369},"inside",{"type":44,"value":1371}," the evaluator function body. For dataset (offline) evaluators, prefer running locally with ",{"type":39,"tag":61,"props":1373,"children":1375},{"className":1374},[],[1376],{"type":44,"value":856},{"type":44,"value":1378}," first — this gives you full package access.",{"type":39,"tag":164,"props":1380,"children":1381},{},[1382],{"type":39,"tag":223,"props":1383,"children":1384},{},[1385],{"type":44,"value":1386},"IMPORTANT - Code vs Structured Evaluators:",{"type":39,"tag":442,"props":1388,"children":1389},{},[1390,1400],{"type":39,"tag":385,"props":1391,"children":1392},{},[1393,1398],{"type":39,"tag":223,"props":1394,"children":1395},{},[1396],{"type":44,"value":1397},"Code evaluators",{"type":44,"value":1399}," (what the CLI uploads): Run in a limited environment without external packages. Use for deterministic logic (exact match, trajectory validation).",{"type":39,"tag":385,"props":1401,"children":1402},{},[1403,1408],{"type":39,"tag":223,"props":1404,"children":1405},{},[1406],{"type":44,"value":1407},"Structured evaluators",{"type":44,"value":1409}," (LLM-as-Judge): Configured via LangSmith UI, use a specific payload format with model\u002Fprompt\u002Fschema. The CLI does not support this format yet.",{"type":39,"tag":164,"props":1411,"children":1412},{},[1413],{"type":39,"tag":223,"props":1414,"children":1415},{},[1416],{"type":44,"value":1417},"IMPORTANT - Choose the right target:",{"type":39,"tag":442,"props":1419,"children":1420},{},[1421,1439],{"type":39,"tag":385,"props":1422,"children":1423},{},[1424,1430,1432,1437],{"type":39,"tag":61,"props":1425,"children":1427},{"className":1426},[],[1428],{"type":44,"value":1429},"--dataset",{"type":44,"value":1431},": Offline evaluator with ",{"type":39,"tag":61,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":44,"value":455},{"type":44,"value":1438}," signature - for comparing to expected values",{"type":39,"tag":385,"props":1440,"children":1441},{},[1442,1448,1450,1455],{"type":39,"tag":61,"props":1443,"children":1445},{"className":1444},[],[1446],{"type":44,"value":1447},"--project",{"type":44,"value":1449},": Online evaluator with ",{"type":39,"tag":61,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":44,"value":496},{"type":44,"value":1456}," signature - for real-time quality checks",{"type":39,"tag":164,"props":1458,"children":1459},{},[1460],{"type":44,"value":1461},"You must specify one. Global evaluators are not supported.",{"type":39,"tag":53,"props":1463,"children":1465},{"className":55,"code":1464,"language":57,"meta":58,"style":58},"# List all evaluators\nlangsmith evaluator list --api-key $LANGSMITH_API_KEY\n\n# Upload offline evaluator (attached to dataset)\nlangsmith evaluator upload my_evaluators.py \\\n  --name \"Trajectory Match\" --function trajectory_evaluator \\\n  --dataset \"My Dataset\" --replace --api-key $LANGSMITH_API_KEY\n\n# Upload online evaluator (attached to project)\nlangsmith evaluator upload my_evaluators.py \\\n  --name \"Quality Check\" --function quality_check \\\n  --project \"Production Agent\" --replace --api-key $LANGSMITH_API_KEY\n\n# Delete\nlangsmith evaluator delete \"Trajectory Match\" --api-key $LANGSMITH_API_KEY\n",[1466],{"type":39,"tag":61,"props":1467,"children":1468},{"__ignoreMap":58},[1469,1477,1500,1507,1515,1541,1578,1612,1619,1627,1650,1683,1716,1723,1731],{"type":39,"tag":65,"props":1470,"children":1471},{"class":67,"line":68},[1472],{"type":39,"tag":65,"props":1473,"children":1474},{"style":90},[1475],{"type":44,"value":1476},"# List all evaluators\n",{"type":39,"tag":65,"props":1478,"children":1479},{"class":67,"line":96},[1480,1484,1488,1492,1496],{"type":39,"tag":65,"props":1481,"children":1482},{"style":196},[1483],{"type":44,"value":22},{"type":39,"tag":65,"props":1485,"children":1486},{"style":84},[1487],{"type":44,"value":203},{"type":39,"tag":65,"props":1489,"children":1490},{"style":84},[1491],{"type":44,"value":208},{"type":39,"tag":65,"props":1493,"children":1494},{"style":84},[1495],{"type":44,"value":213},{"type":39,"tag":65,"props":1497,"children":1498},{"style":72},[1499],{"type":44,"value":218},{"type":39,"tag":65,"props":1501,"children":1502},{"class":67,"line":119},[1503],{"type":39,"tag":65,"props":1504,"children":1505},{"emptyLinePlaceholder":1121},[1506],{"type":44,"value":1124},{"type":39,"tag":65,"props":1508,"children":1509},{"class":67,"line":142},[1510],{"type":39,"tag":65,"props":1511,"children":1512},{"style":90},[1513],{"type":44,"value":1514},"# Upload offline evaluator (attached to dataset)\n",{"type":39,"tag":65,"props":1516,"children":1517},{"class":67,"line":1143},[1518,1522,1526,1531,1536],{"type":39,"tag":65,"props":1519,"children":1520},{"style":196},[1521],{"type":44,"value":22},{"type":39,"tag":65,"props":1523,"children":1524},{"style":84},[1525],{"type":44,"value":203},{"type":39,"tag":65,"props":1527,"children":1528},{"style":84},[1529],{"type":44,"value":1530}," upload",{"type":39,"tag":65,"props":1532,"children":1533},{"style":84},[1534],{"type":44,"value":1535}," my_evaluators.py",{"type":39,"tag":65,"props":1537,"children":1538},{"style":72},[1539],{"type":44,"value":1540}," \\\n",{"type":39,"tag":65,"props":1542,"children":1543},{"class":67,"line":1152},[1544,1549,1554,1559,1564,1569,1574],{"type":39,"tag":65,"props":1545,"children":1546},{"style":84},[1547],{"type":44,"value":1548},"  --name",{"type":39,"tag":65,"props":1550,"children":1551},{"style":78},[1552],{"type":44,"value":1553}," \"",{"type":39,"tag":65,"props":1555,"children":1556},{"style":84},[1557],{"type":44,"value":1558},"Trajectory Match",{"type":39,"tag":65,"props":1560,"children":1561},{"style":78},[1562],{"type":44,"value":1563},"\"",{"type":39,"tag":65,"props":1565,"children":1566},{"style":84},[1567],{"type":44,"value":1568}," --function",{"type":39,"tag":65,"props":1570,"children":1571},{"style":84},[1572],{"type":44,"value":1573}," trajectory_evaluator",{"type":39,"tag":65,"props":1575,"children":1576},{"style":72},[1577],{"type":44,"value":1540},{"type":39,"tag":65,"props":1579,"children":1580},{"class":67,"line":1161},[1581,1586,1590,1595,1599,1604,1608],{"type":39,"tag":65,"props":1582,"children":1583},{"style":84},[1584],{"type":44,"value":1585},"  --dataset",{"type":39,"tag":65,"props":1587,"children":1588},{"style":78},[1589],{"type":44,"value":1553},{"type":39,"tag":65,"props":1591,"children":1592},{"style":84},[1593],{"type":44,"value":1594},"My Dataset",{"type":39,"tag":65,"props":1596,"children":1597},{"style":78},[1598],{"type":44,"value":1563},{"type":39,"tag":65,"props":1600,"children":1601},{"style":84},[1602],{"type":44,"value":1603}," --replace",{"type":39,"tag":65,"props":1605,"children":1606},{"style":84},[1607],{"type":44,"value":213},{"type":39,"tag":65,"props":1609,"children":1610},{"style":72},[1611],{"type":44,"value":218},{"type":39,"tag":65,"props":1613,"children":1614},{"class":67,"line":1169},[1615],{"type":39,"tag":65,"props":1616,"children":1617},{"emptyLinePlaceholder":1121},[1618],{"type":44,"value":1124},{"type":39,"tag":65,"props":1620,"children":1621},{"class":67,"line":1178},[1622],{"type":39,"tag":65,"props":1623,"children":1624},{"style":90},[1625],{"type":44,"value":1626},"# Upload online evaluator (attached to project)\n",{"type":39,"tag":65,"props":1628,"children":1629},{"class":67,"line":1187},[1630,1634,1638,1642,1646],{"type":39,"tag":65,"props":1631,"children":1632},{"style":196},[1633],{"type":44,"value":22},{"type":39,"tag":65,"props":1635,"children":1636},{"style":84},[1637],{"type":44,"value":203},{"type":39,"tag":65,"props":1639,"children":1640},{"style":84},[1641],{"type":44,"value":1530},{"type":39,"tag":65,"props":1643,"children":1644},{"style":84},[1645],{"type":44,"value":1535},{"type":39,"tag":65,"props":1647,"children":1648},{"style":72},[1649],{"type":44,"value":1540},{"type":39,"tag":65,"props":1651,"children":1652},{"class":67,"line":1196},[1653,1657,1661,1666,1670,1674,1679],{"type":39,"tag":65,"props":1654,"children":1655},{"style":84},[1656],{"type":44,"value":1548},{"type":39,"tag":65,"props":1658,"children":1659},{"style":78},[1660],{"type":44,"value":1553},{"type":39,"tag":65,"props":1662,"children":1663},{"style":84},[1664],{"type":44,"value":1665},"Quality Check",{"type":39,"tag":65,"props":1667,"children":1668},{"style":78},[1669],{"type":44,"value":1563},{"type":39,"tag":65,"props":1671,"children":1672},{"style":84},[1673],{"type":44,"value":1568},{"type":39,"tag":65,"props":1675,"children":1676},{"style":84},[1677],{"type":44,"value":1678}," quality_check",{"type":39,"tag":65,"props":1680,"children":1681},{"style":72},[1682],{"type":44,"value":1540},{"type":39,"tag":65,"props":1684,"children":1685},{"class":67,"line":1204},[1686,1691,1695,1700,1704,1708,1712],{"type":39,"tag":65,"props":1687,"children":1688},{"style":84},[1689],{"type":44,"value":1690},"  --project",{"type":39,"tag":65,"props":1692,"children":1693},{"style":78},[1694],{"type":44,"value":1553},{"type":39,"tag":65,"props":1696,"children":1697},{"style":84},[1698],{"type":44,"value":1699},"Production Agent",{"type":39,"tag":65,"props":1701,"children":1702},{"style":78},[1703],{"type":44,"value":1563},{"type":39,"tag":65,"props":1705,"children":1706},{"style":84},[1707],{"type":44,"value":1603},{"type":39,"tag":65,"props":1709,"children":1710},{"style":84},[1711],{"type":44,"value":213},{"type":39,"tag":65,"props":1713,"children":1714},{"style":72},[1715],{"type":44,"value":218},{"type":39,"tag":65,"props":1717,"children":1718},{"class":67,"line":1213},[1719],{"type":39,"tag":65,"props":1720,"children":1721},{"emptyLinePlaceholder":1121},[1722],{"type":44,"value":1124},{"type":39,"tag":65,"props":1724,"children":1725},{"class":67,"line":1222},[1726],{"type":39,"tag":65,"props":1727,"children":1728},{"style":90},[1729],{"type":44,"value":1730},"# Delete\n",{"type":39,"tag":65,"props":1732,"children":1733},{"class":67,"line":1230},[1734,1738,1742,1747,1751,1755,1759,1763],{"type":39,"tag":65,"props":1735,"children":1736},{"style":196},[1737],{"type":44,"value":22},{"type":39,"tag":65,"props":1739,"children":1740},{"style":84},[1741],{"type":44,"value":203},{"type":39,"tag":65,"props":1743,"children":1744},{"style":84},[1745],{"type":44,"value":1746}," delete",{"type":39,"tag":65,"props":1748,"children":1749},{"style":78},[1750],{"type":44,"value":1553},{"type":39,"tag":65,"props":1752,"children":1753},{"style":84},[1754],{"type":44,"value":1558},{"type":39,"tag":65,"props":1756,"children":1757},{"style":78},[1758],{"type":44,"value":1563},{"type":39,"tag":65,"props":1760,"children":1761},{"style":84},[1762],{"type":44,"value":213},{"type":39,"tag":65,"props":1764,"children":1765},{"style":72},[1766],{"type":44,"value":218},{"type":39,"tag":164,"props":1768,"children":1769},{},[1770],{"type":39,"tag":223,"props":1771,"children":1772},{},[1773],{"type":44,"value":1774},"IMPORTANT - Safety Prompts:",{"type":39,"tag":442,"props":1776,"children":1777},{},[1778,1783],{"type":39,"tag":385,"props":1779,"children":1780},{},[1781],{"type":44,"value":1782},"The CLI prompts for confirmation before destructive operations",{"type":39,"tag":385,"props":1784,"children":1785},{},[1786],{"type":39,"tag":223,"props":1787,"children":1788},{},[1789,1791,1797],{"type":44,"value":1790},"NEVER use ",{"type":39,"tag":61,"props":1792,"children":1794},{"className":1793},[],[1795],{"type":44,"value":1796},"--yes",{"type":44,"value":1798}," flag unless the user explicitly requests it",{"type":39,"tag":164,"props":1800,"children":1801},{},[1802],{"type":44,"value":1803},"\u003Cbest_practices>",{"type":39,"tag":381,"props":1805,"children":1806},{},[1807,1817,1838,1848,1858],{"type":39,"tag":385,"props":1808,"children":1809},{},[1810,1815],{"type":39,"tag":223,"props":1811,"children":1812},{},[1813],{"type":44,"value":1814},"Use structured output for LLM judges",{"type":44,"value":1816}," - More reliable than parsing free-text",{"type":39,"tag":385,"props":1818,"children":1819},{},[1820,1825],{"type":39,"tag":223,"props":1821,"children":1822},{},[1823],{"type":44,"value":1824},"Match evaluator to dataset type",{"type":39,"tag":442,"props":1826,"children":1827},{},[1828,1833],{"type":39,"tag":385,"props":1829,"children":1830},{},[1831],{"type":44,"value":1832},"Final Response → LLM as Judge for quality",{"type":39,"tag":385,"props":1834,"children":1835},{},[1836],{"type":44,"value":1837},"Trajectory → Custom Code for sequence",{"type":39,"tag":385,"props":1839,"children":1840},{},[1841,1846],{"type":39,"tag":223,"props":1842,"children":1843},{},[1844],{"type":44,"value":1845},"Use async for LLM judges",{"type":44,"value":1847}," - Enables parallel evaluation",{"type":39,"tag":385,"props":1849,"children":1850},{},[1851,1856],{"type":39,"tag":223,"props":1852,"children":1853},{},[1854],{"type":44,"value":1855},"Test evaluators independently",{"type":44,"value":1857}," - Validate on known good\u002Fbad examples first",{"type":39,"tag":385,"props":1859,"children":1860},{},[1861,1866],{"type":39,"tag":223,"props":1862,"children":1863},{},[1864],{"type":44,"value":1865},"Choose the right language",{"type":39,"tag":442,"props":1867,"children":1868},{},[1869,1874],{"type":39,"tag":385,"props":1870,"children":1871},{},[1872],{"type":44,"value":1873},"Python: Use for Python agents, langchain integrations",{"type":39,"tag":385,"props":1875,"children":1876},{},[1877],{"type":44,"value":1878},"JavaScript: Use for TypeScript\u002FNode.js agents\n\u003C\u002Fbest_practices>",{"type":39,"tag":164,"props":1880,"children":1881},{},[1882],{"type":44,"value":1883},"\u003Crunning_evaluations>",{"type":39,"tag":364,"props":1885,"children":1887},{"id":1886},"running-evaluations",[1888],{"type":44,"value":1889},"Running Evaluations",{"type":39,"tag":164,"props":1891,"children":1892},{},[1893,1898,1900,1905],{"type":39,"tag":223,"props":1894,"children":1895},{},[1896],{"type":44,"value":1897},"Uploaded evaluators",{"type":44,"value":1899}," auto-run when you run experiments - no code needed. ",{"type":39,"tag":223,"props":1901,"children":1902},{},[1903],{"type":44,"value":1904},"Local evaluators",{"type":44,"value":1906}," are passed directly for development\u002Ftesting.",{"type":39,"tag":860,"props":1908,"children":1909},{},[1910,1912,1919,1924,1930,1942,1951],{"type":44,"value":1911},"\n```python\nfrom langsmith import evaluate\n",{"type":39,"tag":1913,"props":1914,"children":1916},"h1",{"id":1915},"uploaded-evaluators-run-automatically",[1917],{"type":44,"value":1918},"Uploaded evaluators run automatically",{"type":39,"tag":164,"props":1920,"children":1921},{},[1922],{"type":44,"value":1923},"results = evaluate(run_agent, data=\"My Dataset\", experiment_prefix=\"eval-v1\")",{"type":39,"tag":1913,"props":1925,"children":1927},{"id":1926},"or-pass-local-evaluators-for-testing",[1928],{"type":44,"value":1929},"Or pass local evaluators for testing",{"type":39,"tag":164,"props":1931,"children":1932},{},[1933,1935,1940],{"type":44,"value":1934},"results = evaluate(run_agent, data=\"My Dataset\", evaluators=",{"type":39,"tag":65,"props":1936,"children":1937},{},[1938],{"type":44,"value":1939},"my_evaluator",{"type":44,"value":1941},", experiment_prefix=\"eval-v1\")",{"type":39,"tag":53,"props":1943,"children":1946},{"className":1944,"code":1945,"language":44},[916],"\u003C\u002Fpython>\n\n\u003Ctypescript>\n```javascript\nimport { evaluate } from \"langsmith\u002Fevaluation\";\n\n\u002F\u002F Uploaded evaluators run automatically\nconst results = await evaluate(runAgent, {\n  data: \"My Dataset\",\n  experimentPrefix: \"eval-v1\",\n});\n\n\u002F\u002F Or pass local evaluators for testing\nconst results = await evaluate(runAgent, {\n  data: \"My Dataset\",\n  evaluators: [myEvaluator],\n  experimentPrefix: \"eval-v1\",\n});\n",[1947],{"type":39,"tag":61,"props":1948,"children":1949},{"__ignoreMap":58},[1950],{"type":44,"value":1945},{"type":39,"tag":1952,"props":1953,"children":1954},"troubleshooting",{},[1955,1957,1967,1984,2001,2032,2046,2056],{"type":44,"value":1956},"\n## Common Issues\n",{"type":39,"tag":164,"props":1958,"children":1959},{},[1960,1965],{"type":39,"tag":223,"props":1961,"children":1962},{},[1963],{"type":44,"value":1964},"Output doesn't match what you expect:",{"type":44,"value":1966}," Query the LangSmith trace. It shows exact inputs\u002Foutputs at each step - compare what you find to what you're trying to extract.",{"type":39,"tag":164,"props":1968,"children":1969},{},[1970,1975,1977,1982],{"type":39,"tag":223,"props":1971,"children":1972},{},[1973],{"type":44,"value":1974},"One metric per evaluator:",{"type":44,"value":1976}," Return ",{"type":39,"tag":61,"props":1978,"children":1980},{"className":1979},[],[1981],{"type":44,"value":750},{"type":44,"value":1983},". For multiple metrics, create separate functions.",{"type":39,"tag":164,"props":1985,"children":1986},{},[1987,1992,1994,2000],{"type":39,"tag":223,"props":1988,"children":1989},{},[1990],{"type":44,"value":1991},"Field name mismatch:",{"type":44,"value":1993}," Your run function output must match dataset schema exactly. Inspect dataset first with ",{"type":39,"tag":61,"props":1995,"children":1997},{"className":1996},[],[1998],{"type":44,"value":1999},"client.read_example(example_id)",{"type":44,"value":858},{"type":39,"tag":164,"props":2002,"children":2003},{},[2004,2009,2011,2016,2018,2023,2025,2030],{"type":39,"tag":223,"props":2005,"children":2006},{},[2007],{"type":44,"value":2008},"RunTree vs dict (Python only):",{"type":44,"value":2010}," Local ",{"type":39,"tag":61,"props":2012,"children":2014},{"className":2013},[],[2015],{"type":44,"value":584},{"type":44,"value":2017}," passes ",{"type":39,"tag":61,"props":2019,"children":2021},{"className":2020},[],[2022],{"type":44,"value":658},{"type":44,"value":2024},", uploaded evaluators receive ",{"type":39,"tag":61,"props":2026,"children":2028},{"className":2027},[],[2029],{"type":44,"value":677},{"type":44,"value":2031},". Handle both:",{"type":39,"tag":53,"props":2033,"children":2035},{"className":1102,"code":2034,"language":860,"meta":58,"style":58},"run_outputs = run.outputs if hasattr(run, \"outputs\") else run.get(\"outputs\", {}) or {}\n",[2036],{"type":39,"tag":61,"props":2037,"children":2038},{"__ignoreMap":58},[2039],{"type":39,"tag":65,"props":2040,"children":2041},{"class":67,"line":68},[2042],{"type":39,"tag":65,"props":2043,"children":2044},{},[2045],{"type":44,"value":2034},{"type":39,"tag":164,"props":2047,"children":2048},{},[2049,2051],{"type":44,"value":2050},"TypeScript always uses attribute access: ",{"type":39,"tag":61,"props":2052,"children":2054},{"className":2053},[],[2055],{"type":44,"value":721},{"type":39,"tag":2057,"props":2058,"children":2059},"resources",{},[2060],{"type":44,"value":2061},"\n- [LangSmith Evaluation Concepts](https:\u002F\u002Fdocs.langchain.com\u002Flangsmith\u002Fevaluation-concepts)\n- [Custom Code Evaluators](https:\u002F\u002Fchangelog.langchain.com\u002Fannouncements\u002Fcustom-code-evaluators-in-langsmith)\n- [OpenEvals - Readymade Evaluators](https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenevals)\n",{"type":39,"tag":2063,"props":2064,"children":2065},"style",{},[2066],{"type":44,"value":2067},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2069,"total":119},[2070,2083,2089],{"slug":2071,"name":2071,"fn":2072,"description":2073,"org":2074,"tags":2075,"stars":23,"repoUrl":24,"updatedAt":2082},"langsmith-dataset","create and manage LangSmith eval datasets","INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets. Covers dataset types (final_response, single_step, trajectory, RAG), CLI management commands, SDK-based creation, and example management. Uses the langsmith CLI tool.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2076,2079,2080,2081],{"name":2077,"slug":2078,"type":16},"Datasets","datasets",{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},"2026-04-11T04:40:26.848233",{"slug":4,"name":4,"fn":5,"description":6,"org":2084,"tags":2085,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2086,2087,2088],{"name":18,"slug":19,"type":16},{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"slug":2090,"name":2090,"fn":2091,"description":2092,"org":2093,"tags":2094,"stars":23,"repoUrl":24,"updatedAt":2102},"langsmith-trace","add tracing and query LangSmith traces","INVOKE THIS SKILL when working with LangSmith tracing OR querying traces. Covers adding tracing to applications and querying\u002Fexporting trace data. Uses the langsmith CLI tool.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2095,2096,2099],{"name":21,"slug":22,"type":16},{"name":2097,"slug":2098,"type":16},"Observability","observability",{"name":2100,"slug":2101,"type":16},"Tracing","tracing","2026-04-11T04:40:29.299419",{"items":2104,"total":2283},[2105,2126,2137,2154,2167,2184,2201,2216,2230,2240,2251,2270],{"slug":2106,"name":2106,"fn":2107,"description":2108,"org":2109,"tags":2110,"stars":2123,"repoUrl":2124,"updatedAt":2125},"analyze-market","perform market analysis and size estimation","Perform a market analysis for a product category or segment. Trigger on: market analysis, market size, TAM SAM SOM, market opportunity, industry analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2111,2114,2117,2120],{"name":2112,"slug":2113,"type":16},"Marketing","marketing",{"name":2115,"slug":2116,"type":16},"Research","research",{"name":2118,"slug":2119,"type":16},"Sales","sales",{"name":2121,"slug":2122,"type":16},"Strategy","strategy",26592,"https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fdeepagents","2026-04-18T04:46:54.557115",{"slug":2127,"name":2127,"fn":2128,"description":2129,"org":2130,"tags":2131,"stars":2123,"repoUrl":2124,"updatedAt":2136},"arxiv-search","search arXiv for academic research papers","Searches arXiv for preprints and academic papers, retrieves abstracts, and filters by topic. Use when the user asks to find research papers, search arXiv, look up preprints, find academic articles in physics, math, CS, biology, statistics, or related fields.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2132,2133],{"name":2115,"slug":2116,"type":16},{"name":2134,"slug":2135,"type":16},"Search","search","2026-05-13T06:11:01.203061",{"slug":2138,"name":2138,"fn":2139,"description":2140,"org":2141,"tags":2142,"stars":2123,"repoUrl":2124,"updatedAt":2153},"blog-post","write SEO-optimized blog posts","Write long-form blog posts with SEO optimization and clear structure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2143,2146,2147,2150],{"name":2144,"slug":2145,"type":16},"Content Creation","content-creation",{"name":2112,"slug":2113,"type":16},{"name":2148,"slug":2149,"type":16},"SEO","seo",{"name":2151,"slug":2152,"type":16},"Writing","writing","2026-04-15T05:00:54.149813",{"slug":2155,"name":2155,"fn":2156,"description":2157,"org":2158,"tags":2159,"stars":2123,"repoUrl":2124,"updatedAt":2166},"competitor-analysis","analyze competitors and market positioning","Analyze competitors in a given market segment. Trigger on: competitive landscape, competitor analysis, market comparison, competitive positioning.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2160,2163,2164,2165],{"name":2161,"slug":2162,"type":16},"Competitive Intelligence","competitive-intelligence",{"name":2112,"slug":2113,"type":16},{"name":2115,"slug":2116,"type":16},{"name":2121,"slug":2122,"type":16},"2026-04-18T04:46:55.79306",{"slug":2168,"name":2168,"fn":2169,"description":2170,"org":2171,"tags":2172,"stars":2123,"repoUrl":2124,"updatedAt":2183},"deepagents-thread-inspector","inspect local Deep Agents conversation threads","Inspect and explain conversations in the local Deep Agents Code SQLite session store. Use as a fallback when LangSmith trace tooling is unavailable, for offline or untraced sessions, or when asked to identify or summarize a local dcode thread, inspect checkpoint metadata, list recent local threads, or parse ~\u002F.deepagents\u002F.state\u002Fsessions.db and a thread UUID or prefix.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2173,2176,2179,2180],{"name":2174,"slug":2175,"type":16},"Agents","agents",{"name":2177,"slug":2178,"type":16},"Debugging","debugging",{"name":9,"slug":8,"type":16},{"name":2181,"slug":2182,"type":16},"SQLite","sqlite","2026-07-24T06:08:57.102689",{"slug":2185,"name":2185,"fn":2186,"description":2187,"org":2188,"tags":2189,"stars":2123,"repoUrl":2124,"updatedAt":2200},"langgraph-docs","build stateful agents with LangGraph","Fetches and references LangGraph Python documentation to build stateful agents, create multi-agent workflows, and implement human-in-the-loop patterns. Use when the user asks about LangGraph, graph agents, state machines, agent orchestration, LangGraph API, or needs LangGraph implementation guidance.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2190,2191,2194,2197],{"name":2174,"slug":2175,"type":16},{"name":2192,"slug":2193,"type":16},"Documentation","documentation",{"name":2195,"slug":2196,"type":16},"LangGraph","langgraph",{"name":2198,"slug":2199,"type":16},"Multi-Agent","multi-agent","2026-05-13T06:11:03.650877",{"slug":2202,"name":2202,"fn":2203,"description":2204,"org":2205,"tags":2206,"stars":2123,"repoUrl":2124,"updatedAt":2215},"remember","capture knowledge into persistent memory","Review the current conversation and capture valuable knowledge — best practices, coding conventions, architecture decisions, workflows, and user feedback — into persistent memory (AGENTS.md) or reusable skills. Use when the user says: (1) remember this, (2) save what we learned, (3) update memory, (4) capture learnings.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2207,2208,2209,2212],{"name":2174,"slug":2175,"type":16},{"name":2192,"slug":2193,"type":16},{"name":2210,"slug":2211,"type":16},"Knowledge Management","knowledge-management",{"name":2213,"slug":2214,"type":16},"Memory","memory","2026-05-13T06:10:58.510037",{"slug":2217,"name":2217,"fn":2218,"description":2219,"org":2220,"tags":2221,"stars":2123,"repoUrl":2124,"updatedAt":2229},"skill-creator","create agent skills and tool integrations","Guide for creating effective skills that extend agent capabilities with specialized knowledge, workflows, or tool integrations. Use this skill when the user asks to: (1) create a new skill, (2) make a skill, (3) build a skill, (4) set up a skill, (5) initialize a skill, (6) scaffold a skill, (7) update or modify an existing skill, (8) validate a skill, (9) learn about skill structure, (10) understand how skills work, or (11) get guidance on skill design patterns. Trigger on phrases like \"create a skill\", \"new skill\", \"make a skill\", \"skill for X\", \"how do I create a skill\", or \"help me build a skill\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2222,2223,2226],{"name":2174,"slug":2175,"type":16},{"name":2224,"slug":2225,"type":16},"Engineering","engineering",{"name":2227,"slug":2228,"type":16},"Plugin Development","plugin-development","2026-05-13T06:10:59.88449",{"slug":2231,"name":2231,"fn":2232,"description":2233,"org":2234,"tags":2235,"stars":2123,"repoUrl":2124,"updatedAt":2239},"social-media","create optimized social media posts","Create social media posts optimized for engagement across platforms.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2236,2237,2238],{"name":2144,"slug":2145,"type":16},{"name":2112,"slug":2113,"type":16},{"name":2151,"slug":2152,"type":16},"2026-04-15T05:00:55.37452",{"slug":2241,"name":2241,"fn":2242,"description":2243,"org":2244,"tags":2245,"stars":2123,"repoUrl":2124,"updatedAt":2250},"web-research","conduct and synthesize web research","Searches multiple web sources, synthesizes findings, and produces cited research reports using delegated subagents. Use when the user asks to research a topic online, search the web, look something up, find current information, compare options, or produce a research report.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2246,2247,2248,2249],{"name":2174,"slug":2175,"type":16},{"name":2198,"slug":2199,"type":16},{"name":2115,"slug":2116,"type":16},{"name":2134,"slug":2135,"type":16},"2026-05-13T06:11:04.930044",{"slug":2252,"name":2252,"fn":2253,"description":2254,"org":2255,"tags":2256,"stars":2267,"repoUrl":2268,"updatedAt":2269},"mermaid-diagrams","embed Mermaid diagrams in documentation","Embed Mermaid diagrams in generated wiki pages. Use whenever documenting a runtime or request flow, a call sequence, a state machine or lifecycle, a data model or entity relationships, or non-trivial control flow, since these are clearer as a diagram than as prose. Also use when an update run touches a page that already contains a mermaid fence, or a page that contains a text fence a previous run degraded.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2257,2260,2261,2264],{"name":2258,"slug":2259,"type":16},"Diagrams","diagrams",{"name":2192,"slug":2193,"type":16},{"name":2262,"slug":2263,"type":16},"Markdown","markdown",{"name":2265,"slug":2266,"type":16},"Technical Writing","technical-writing",12181,"https:\u002F\u002Fgithub.com\u002Flangchain-ai\u002Fopenwiki","2026-07-24T06:09:01.089597",{"slug":2271,"name":2271,"fn":2272,"description":2273,"org":2274,"tags":2275,"stars":2267,"repoUrl":2268,"updatedAt":2282},"write-connector","implement OpenWiki source connectors","Add a new built-in OpenWiki source connector. Use when a user asks to create or implement an OpenWiki connector.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2276,2279],{"name":2277,"slug":2278,"type":16},"API Development","api-development",{"name":2280,"slug":2281,"type":16},"Integrations","integrations","2026-07-18T05:48:23.961804",41]