[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-mlflow-analyzing-mlflow-session":3,"mdc-skhn7-key":33,"related-org-mlflow-analyzing-mlflow-session":1856,"related-repo-mlflow-analyzing-mlflow-session":2018},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"analyzing-mlflow-session","analyze MLflow chat conversation sessions","Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction. Use when the user asks to debug a chat conversation, review session or chat history, find where a multi-turn chat went wrong, or analyze patterns across turns. Triggers on \"analyze this session\", \"what happened in this conversation\", \"debug session\", \"review chat history\", \"where did this chat go wrong\", \"session traces\", \"analyze chat\", \"debug this chat\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"mlflow","MLflow","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmlflow.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Tracing","tracing","tag",{"name":17,"slug":18,"type":15},"Observability","observability",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Debugging","debugging",60,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills","2026-07-14T05:39:10.542342",null,19,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills\u002Ftree\u002FHEAD\u002Fanalyze-mlflow-chat-session","---\nname: analyzing-mlflow-session\ndescription: Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction. Use when the user asks to debug a chat conversation, review session or chat history, find where a multi-turn chat went wrong, or analyze patterns across turns. Triggers on \"analyze this session\", \"what happened in this conversation\", \"debug session\", \"review chat history\", \"where did this chat go wrong\", \"session traces\", \"analyze chat\", \"debug this chat\".\n---\n\n# Analyzing an MLflow Chat Session\n\n## What is a Session?\n\nA session groups multiple traces that belong to the same chat conversation or user interaction. Each trace in the session represents one turn: the user's input and the system's response. Traces within a session are linked by a shared session ID stored in trace metadata.\n\nThe session ID is stored in trace metadata under the key `mlflow.trace.session`. This key contains dots, which affects filter syntax (see below). All traces sharing the same value for this key belong to the same session.\n\n## Reconstructing the Conversation\n\nReconstructing a session's conversation is a multi-step process: discover the input\u002Foutput schema from the first trace, extract those fields efficiently across all session traces, then inspect specific turns as needed. **Do NOT fetch full traces for every turn** — use `--extract-fields` on the search command instead.\n\n**Step 1: Discover the schema.** First, find a trace ID from the session, then fetch its full JSON to inspect the schema:\n\n```bash\n# Get the first trace in the session\nmlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  --filter-string 'metadata.`mlflow.trace.session` = \"\u003CSESSION_ID>\"' \\\n  --order-by \"timestamp_ms ASC\" \\\n  --extract-fields 'info.trace_id' \\\n  --output json \\\n  --max-results 1 > \u002Ftmp\u002Ffirst_trace.json\n\n# Fetch the full trace (always outputs JSON, no --output flag needed)\nmlflow traces get \\\n  --trace-id \u003CTRACE_ID_FROM_ABOVE> > \u002Ftmp\u002Ftrace_detail.json\n```\n\nFind the **root span** — the span with `parent_span_id` equal to `null` (i.e., it has no parent). This is the top-level operation in the trace:\n\n```bash\n# Find the root span\njq '.data.spans[] | select(.parent_span_id == null)' \u002Ftmp\u002Ftrace_detail.json\n```\n\nExamine its `attributes` dict to identify which keys hold the user input and system output. These could be:\n\n- **MLflow standard attributes**: `mlflow.spanInputs` and `mlflow.spanOutputs` (set by the MLflow Python client)\n- **Custom attributes**: Application-specific keys set via `@mlflow.trace` or `mlflow.start_span()` with custom attribute logging\n- **Third-party OTel attributes**: Keys following GenAI Semantic Conventions, OpenInference, or other instrumentation conventions\n\nThe structure of these values also varies by application (e.g., a `query` string, a `messages` array, a dict with multiple fields). Inspect the actual attribute values to understand the format.\n\n**If the root span has empty or missing inputs\u002Foutputs**, it may be a wrapper span (e.g., an orchestrator or middleware) that doesn't directly carry the chat turn data. In that case, look at its immediate children — find the closest span to the top of the hierarchy that has meaningful inputs and outputs corresponding to a chat turn:\n\nThe following example assumes the trace comes from the MLflow Python client (which stores inputs\u002Foutputs in `mlflow.spanInputs`\u002F`mlflow.spanOutputs`) and that the relevant span is a direct child of root. In practice, the relevant span may be deeper in the hierarchy, and traces from other clients may use different attribute keys — explore the span tree as needed:\n\n```bash\n# Get the root span's ID\nROOT_ID=$(jq -r '.data.spans[] | select(.parent_span_id == null) | .span_id' \u002Ftmp\u002Ftrace_detail.json)\n\n# List immediate children of the root span with their inputs\u002Foutputs\njq --arg root \"$ROOT_ID\" '.data.spans[] | select(.parent_span_id == $root) | {name: .name, inputs: .attributes[\"mlflow.spanInputs\"], outputs: .attributes[\"mlflow.spanOutputs\"]}' \u002Ftmp\u002Ftrace_detail.json\n```\n\nAlso check the first trace's assessments. **Session-level assessments are attached to the first trace in the session** — these evaluate the session as a whole (e.g., overall conversation quality, multi-turn coherence) and can indicate the presence of issues somewhere across the entire session, not just the first turn. The first trace may also have per-turn assessments for that specific turn.\n\nBoth types appear in `.info.assessments`. Session-level assessments are identified by the presence of `mlflow.trace.session` in their `metadata` field:\n\n```bash\n# Show session-level assessments (exclude scorer errors)\njq '[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"]) | {name: .assessment_name, value: .feedback.value}]' \u002Ftmp\u002Ftrace_detail.json\n\n# Show per-turn assessments (exclude scorer errors)\njq '[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"] == null) | {name: .assessment_name, value: .feedback.value}]' \u002Ftmp\u002Ftrace_detail.json\n```\n\n**Assessment errors are not trace errors.** If an assessment has a `feedback.error` field, it means the scorer or judge failed — not that the trace itself has a problem. Exclude these when using assessments to identify trace issues.\n\n**Always consult the rationale when interpreting assessment values.** The `value` alone can be misleading — for example, a `user_frustration` assessment with `value: \"no\"` could mean \"no frustration detected\" or \"the frustration check did not pass\" (i.e., frustration *is* present), depending on how the scorer was configured. The `.rationale` field (a top-level assessment field, **not** nested under `.feedback`) explains what the value means in context. Include rationale when extracting assessments:\n\n```bash\njq '[.info.assessments[] | select(.feedback.error == null) | {name: .assessment_name, value: .feedback.value, rationale: .rationale}]' \u002Ftmp\u002Ftrace_detail.json\n```\n\n**Step 2: Extract across all session traces.** Once you know which attribute keys hold inputs and outputs, search for all traces in the session using `--extract-fields` to pull those fields along with assessments (see [Handling CLI Output](#handling-cli-output) for why output is written to a file):\n\n```bash\nmlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  --filter-string 'metadata.`mlflow.trace.session` = \"\u003CSESSION_ID>\"' \\\n  --order-by \"timestamp_ms ASC\" \\\n  --extract-fields 'info.trace_id,info.state,info.request_time,info.assessments,info.trace_metadata.`mlflow.traceInputs`,info.trace_metadata.`mlflow.traceOutputs`' \\\n  --output json \\\n  --max-results 100 > \u002Ftmp\u002Fsession_traces.json\n```\n\nThen use bash commands (e.g., `jq`, `wc`, `head`) on the file to analyze it.\n\nThe `--extract-fields` example above uses `mlflow.traceInputs`\u002F`mlflow.traceOutputs` from trace metadata — adjust the field paths based on what you discovered in step 1.\n\nAssessments contain quality judgments (e.g., correctness, relevance) that can pinpoint which turns had issues without needing to read every trace in detail. To identify which turns have assessment signals (excluding scorer errors):\n\n```bash\n# List turns with their valid assessments (scorer errors filtered out)\njq '.traces[] | {\n  trace_id: .info.trace_id,\n  time: .info.request_time,\n  state: .info.state,\n  assessments: [.info.assessments[]? | select(.feedback.error == null) | {\n    name: .assessment_name,\n    value: .feedback.value\n  }]\n}' \u002Ftmp\u002Fsession_traces.json\n```\n\n**CLI syntax notes:**\n\n- **`--experiment-id` is required** for all `mlflow traces search` commands. The command will fail without it.\n- Metadata keys containing dots **must** be escaped with backticks in filter strings and extract-fields: `` metadata.`mlflow.trace.session` ``\n- **Shell quoting**: Backticks inside **double quotes** are interpreted by bash as command substitution (e.g., bash will try to run `` `mlflow.trace.session` `` as a command). Always use **single quotes** for the outer string when the value contains backticks. For example: `--filter-string 'metadata.\\`mlflow.trace.session\\` = \"value\"'`\n- `--max-results` defaults to 100, which is sufficient for most sessions. Increase up to 500 (the maximum) for longer conversations. If 500 results are returned, use pagination to retrieve the rest.\n\n## Handling CLI Output\n\nMLflow trace output can be large, and Claude Code's Bash tool has a ~30KB output limit for piped commands. When output exceeds this threshold, it gets saved to a file instead of being piped, causing silent failures.\n\n**Safe approach (always works):**\n```bash\n# Step 1: Save to file\nmlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  [...] \\\n  --output json > \u002Ftmp\u002Foutput.json\n\n# Step 2: Process the file\ncat \u002Ftmp\u002Foutput.json | jq '.traces[0].info.trace_id'\nhead -50 \u002Ftmp\u002Foutput.json\nwc -l \u002Ftmp\u002Foutput.json\n```\n\n**Never pipe MLflow CLI output directly** (e.g., `mlflow traces search ... | jq '.'`). This can silently produce no output. Always redirect to a file first, then run commands on the file.\n\nTo inspect a specific turn in detail (e.g., after identifying a problematic turn), fetch its full trace:\n\n```bash\nmlflow traces get --trace-id \u003CTRACE_ID> > \u002Ftmp\u002Fturn_detail.json\n```\n\n\n## Codebase Correlation\n\n- **Session ID assignment**: Search the codebase for where `mlflow.trace.session` is set to understand how sessions are created — per user login, per browser tab, per explicit \"new conversation\" action, etc.\n- **Context window management**: Look for how the application constructs the message history passed to the LLM at each turn. Common patterns include sliding window (last N messages), summarization of older turns, or full history. This implementation determines what context the model sees and is a frequent source of multi-turn failures.\n- **Memory and state**: Some applications maintain state across turns beyond message history (e.g., extracted entities, user preferences, accumulated tool results). Search for how this state is stored and passed between turns.\n\n## Reference Scripts\n\nThe `scripts\u002F` subdirectory contains ready-to-run bash scripts for each analysis step. All scripts follow the output handling rules above (redirect to file, then process).\n\n- **`scripts\u002Fdiscover_schema.sh \u003CEXPERIMENT_ID> \u003CSESSION_ID>`** — Finds the first trace in the session, fetches its full detail, and prints the root span's attribute keys and input\u002Foutput values.\n- **`scripts\u002Finspect_turn.sh \u003CTRACE_ID>`** — Fetches a specific trace, lists all spans, highlights error spans, and shows assessments.\n\n## Example: Wrong Answer on Chat Turn 5\n\nA user reports that their chatbot gave an incorrect answer on the 5th message of a chat conversation.\n\n**1. Discover the schema and reconstruct the conversation.**\n\nFetch the first trace in the session and inspect the root span's attributes to find which keys hold inputs and outputs. In this case, `mlflow.spanInputs` contains the user query and `mlflow.spanOutputs` contains the assistant response. Then search all session traces, extracting those fields in chronological order. Scanning the extracted inputs and outputs confirms that turn 5's response is wrong, and reveals whether earlier turns look correct.\n\n**2. Check if the error originated in an earlier turn.**\n\nTurn 3's response contains a factual error that the user didn't challenge. Turn 4 builds on that incorrect information, and turn 5 compounds it. The root cause is in turn 3, not turn 5.\n\n**3. Analyze the root-cause turn as a single trace.**\n\nFetch the full trace for turn 3 and analyze it — examine assessments (if any), walk the span tree, check retriever results, and correlate with code. The retriever returned an outdated document, causing the wrong answer.\n\n**4. Recommendations.**\n\n- Fix the retriever's data source to exclude or update outdated documents.\n- Add per-turn assessments to detect errors before they propagate across the conversation.\n- Consider implementing conversation-level error detection (e.g., checking consistency of answers across turns).\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,47,54,60,74,80,101,111,395,423,463,476,543,564,574,593,715,727,755,833,851,916,947,973,1132,1159,1186,1191,1294,1302,1398,1403,1408,1416,1589,1607,1612,1665,1671,1711,1717,1729,1760,1766,1771,1779,1798,1806,1811,1819,1824,1832,1850],{"type":39,"tag":40,"props":41,"children":43},"element","h1",{"id":42},"analyzing-an-mlflow-chat-session",[44],{"type":45,"value":46},"text","Analyzing an MLflow Chat Session",{"type":39,"tag":48,"props":49,"children":51},"h2",{"id":50},"what-is-a-session",[52],{"type":45,"value":53},"What is a Session?",{"type":39,"tag":55,"props":56,"children":57},"p",{},[58],{"type":45,"value":59},"A session groups multiple traces that belong to the same chat conversation or user interaction. Each trace in the session represents one turn: the user's input and the system's response. Traces within a session are linked by a shared session ID stored in trace metadata.",{"type":39,"tag":55,"props":61,"children":62},{},[63,65,72],{"type":45,"value":64},"The session ID is stored in trace metadata under the key ",{"type":39,"tag":66,"props":67,"children":69},"code",{"className":68},[],[70],{"type":45,"value":71},"mlflow.trace.session",{"type":45,"value":73},". This key contains dots, which affects filter syntax (see below). All traces sharing the same value for this key belong to the same session.",{"type":39,"tag":48,"props":75,"children":77},{"id":76},"reconstructing-the-conversation",[78],{"type":45,"value":79},"Reconstructing the Conversation",{"type":39,"tag":55,"props":81,"children":82},{},[83,85,91,93,99],{"type":45,"value":84},"Reconstructing a session's conversation is a multi-step process: discover the input\u002Foutput schema from the first trace, extract those fields efficiently across all session traces, then inspect specific turns as needed. ",{"type":39,"tag":86,"props":87,"children":88},"strong",{},[89],{"type":45,"value":90},"Do NOT fetch full traces for every turn",{"type":45,"value":92}," — use ",{"type":39,"tag":66,"props":94,"children":96},{"className":95},[],[97],{"type":45,"value":98},"--extract-fields",{"type":45,"value":100}," on the search command instead.",{"type":39,"tag":55,"props":102,"children":103},{},[104,109],{"type":39,"tag":86,"props":105,"children":106},{},[107],{"type":45,"value":108},"Step 1: Discover the schema.",{"type":45,"value":110}," First, find a trace ID from the session, then fetch its full JSON to inspect the schema:",{"type":39,"tag":112,"props":113,"children":118},"pre",{"className":114,"code":115,"language":116,"meta":117,"style":117},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Get the first trace in the session\nmlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  --filter-string 'metadata.`mlflow.trace.session` = \"\u003CSESSION_ID>\"' \\\n  --order-by \"timestamp_ms ASC\" \\\n  --extract-fields 'info.trace_id' \\\n  --output json \\\n  --max-results 1 > \u002Ftmp\u002Ffirst_trace.json\n\n# Fetch the full trace (always outputs JSON, no --output flag needed)\nmlflow traces get \\\n  --trace-id \u003CTRACE_ID_FROM_ABOVE> > \u002Ftmp\u002Ftrace_detail.json\n","bash","",[119],{"type":39,"tag":66,"props":120,"children":121},{"__ignoreMap":117},[122,134,160,194,222,250,276,294,319,329,338,359],{"type":39,"tag":123,"props":124,"children":127},"span",{"class":125,"line":126},"line",1,[128],{"type":39,"tag":123,"props":129,"children":131},{"style":130},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[132],{"type":45,"value":133},"# Get the first trace in the session\n",{"type":39,"tag":123,"props":135,"children":137},{"class":125,"line":136},2,[138,143,149,154],{"type":39,"tag":123,"props":139,"children":141},{"style":140},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[142],{"type":45,"value":8},{"type":39,"tag":123,"props":144,"children":146},{"style":145},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[147],{"type":45,"value":148}," traces",{"type":39,"tag":123,"props":150,"children":151},{"style":145},[152],{"type":45,"value":153}," search",{"type":39,"tag":123,"props":155,"children":157},{"style":156},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[158],{"type":45,"value":159}," \\\n",{"type":39,"tag":123,"props":161,"children":163},{"class":125,"line":162},3,[164,169,175,180,185,190],{"type":39,"tag":123,"props":165,"children":166},{"style":145},[167],{"type":45,"value":168},"  --experiment-id",{"type":39,"tag":123,"props":170,"children":172},{"style":171},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[173],{"type":45,"value":174}," \u003C",{"type":39,"tag":123,"props":176,"children":177},{"style":145},[178],{"type":45,"value":179},"EXPERIMENT_I",{"type":39,"tag":123,"props":181,"children":182},{"style":156},[183],{"type":45,"value":184},"D",{"type":39,"tag":123,"props":186,"children":187},{"style":171},[188],{"type":45,"value":189},">",{"type":39,"tag":123,"props":191,"children":192},{"style":156},[193],{"type":45,"value":159},{"type":39,"tag":123,"props":195,"children":197},{"class":125,"line":196},4,[198,203,208,213,218],{"type":39,"tag":123,"props":199,"children":200},{"style":145},[201],{"type":45,"value":202},"  --filter-string",{"type":39,"tag":123,"props":204,"children":205},{"style":171},[206],{"type":45,"value":207}," '",{"type":39,"tag":123,"props":209,"children":210},{"style":145},[211],{"type":45,"value":212},"metadata.`mlflow.trace.session` = \"\u003CSESSION_ID>\"",{"type":39,"tag":123,"props":214,"children":215},{"style":171},[216],{"type":45,"value":217},"'",{"type":39,"tag":123,"props":219,"children":220},{"style":156},[221],{"type":45,"value":159},{"type":39,"tag":123,"props":223,"children":225},{"class":125,"line":224},5,[226,231,236,241,246],{"type":39,"tag":123,"props":227,"children":228},{"style":145},[229],{"type":45,"value":230},"  --order-by",{"type":39,"tag":123,"props":232,"children":233},{"style":171},[234],{"type":45,"value":235}," \"",{"type":39,"tag":123,"props":237,"children":238},{"style":145},[239],{"type":45,"value":240},"timestamp_ms ASC",{"type":39,"tag":123,"props":242,"children":243},{"style":171},[244],{"type":45,"value":245},"\"",{"type":39,"tag":123,"props":247,"children":248},{"style":156},[249],{"type":45,"value":159},{"type":39,"tag":123,"props":251,"children":253},{"class":125,"line":252},6,[254,259,263,268,272],{"type":39,"tag":123,"props":255,"children":256},{"style":145},[257],{"type":45,"value":258},"  --extract-fields",{"type":39,"tag":123,"props":260,"children":261},{"style":171},[262],{"type":45,"value":207},{"type":39,"tag":123,"props":264,"children":265},{"style":145},[266],{"type":45,"value":267},"info.trace_id",{"type":39,"tag":123,"props":269,"children":270},{"style":171},[271],{"type":45,"value":217},{"type":39,"tag":123,"props":273,"children":274},{"style":156},[275],{"type":45,"value":159},{"type":39,"tag":123,"props":277,"children":279},{"class":125,"line":278},7,[280,285,290],{"type":39,"tag":123,"props":281,"children":282},{"style":145},[283],{"type":45,"value":284},"  --output",{"type":39,"tag":123,"props":286,"children":287},{"style":145},[288],{"type":45,"value":289}," json",{"type":39,"tag":123,"props":291,"children":292},{"style":156},[293],{"type":45,"value":159},{"type":39,"tag":123,"props":295,"children":297},{"class":125,"line":296},8,[298,303,309,314],{"type":39,"tag":123,"props":299,"children":300},{"style":145},[301],{"type":45,"value":302},"  --max-results",{"type":39,"tag":123,"props":304,"children":306},{"style":305},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[307],{"type":45,"value":308}," 1",{"type":39,"tag":123,"props":310,"children":311},{"style":171},[312],{"type":45,"value":313}," >",{"type":39,"tag":123,"props":315,"children":316},{"style":145},[317],{"type":45,"value":318}," \u002Ftmp\u002Ffirst_trace.json\n",{"type":39,"tag":123,"props":320,"children":322},{"class":125,"line":321},9,[323],{"type":39,"tag":123,"props":324,"children":326},{"emptyLinePlaceholder":325},true,[327],{"type":45,"value":328},"\n",{"type":39,"tag":123,"props":330,"children":332},{"class":125,"line":331},10,[333],{"type":39,"tag":123,"props":334,"children":335},{"style":130},[336],{"type":45,"value":337},"# Fetch the full trace (always outputs JSON, no --output flag needed)\n",{"type":39,"tag":123,"props":339,"children":341},{"class":125,"line":340},11,[342,346,350,355],{"type":39,"tag":123,"props":343,"children":344},{"style":140},[345],{"type":45,"value":8},{"type":39,"tag":123,"props":347,"children":348},{"style":145},[349],{"type":45,"value":148},{"type":39,"tag":123,"props":351,"children":352},{"style":145},[353],{"type":45,"value":354}," get",{"type":39,"tag":123,"props":356,"children":357},{"style":156},[358],{"type":45,"value":159},{"type":39,"tag":123,"props":360,"children":362},{"class":125,"line":361},12,[363,368,372,377,382,386,390],{"type":39,"tag":123,"props":364,"children":365},{"style":145},[366],{"type":45,"value":367},"  --trace-id",{"type":39,"tag":123,"props":369,"children":370},{"style":171},[371],{"type":45,"value":174},{"type":39,"tag":123,"props":373,"children":374},{"style":145},[375],{"type":45,"value":376},"TRACE_ID_FROM_ABOV",{"type":39,"tag":123,"props":378,"children":379},{"style":156},[380],{"type":45,"value":381},"E",{"type":39,"tag":123,"props":383,"children":384},{"style":171},[385],{"type":45,"value":189},{"type":39,"tag":123,"props":387,"children":388},{"style":171},[389],{"type":45,"value":313},{"type":39,"tag":123,"props":391,"children":392},{"style":145},[393],{"type":45,"value":394}," \u002Ftmp\u002Ftrace_detail.json\n",{"type":39,"tag":55,"props":396,"children":397},{},[398,400,405,407,413,415,421],{"type":45,"value":399},"Find the ",{"type":39,"tag":86,"props":401,"children":402},{},[403],{"type":45,"value":404},"root span",{"type":45,"value":406}," — the span with ",{"type":39,"tag":66,"props":408,"children":410},{"className":409},[],[411],{"type":45,"value":412},"parent_span_id",{"type":45,"value":414}," equal to ",{"type":39,"tag":66,"props":416,"children":418},{"className":417},[],[419],{"type":45,"value":420},"null",{"type":45,"value":422}," (i.e., it has no parent). This is the top-level operation in the trace:",{"type":39,"tag":112,"props":424,"children":426},{"className":114,"code":425,"language":116,"meta":117,"style":117},"# Find the root span\njq '.data.spans[] | select(.parent_span_id == null)' \u002Ftmp\u002Ftrace_detail.json\n",[427],{"type":39,"tag":66,"props":428,"children":429},{"__ignoreMap":117},[430,438],{"type":39,"tag":123,"props":431,"children":432},{"class":125,"line":126},[433],{"type":39,"tag":123,"props":434,"children":435},{"style":130},[436],{"type":45,"value":437},"# Find the root span\n",{"type":39,"tag":123,"props":439,"children":440},{"class":125,"line":136},[441,446,450,455,459],{"type":39,"tag":123,"props":442,"children":443},{"style":140},[444],{"type":45,"value":445},"jq",{"type":39,"tag":123,"props":447,"children":448},{"style":171},[449],{"type":45,"value":207},{"type":39,"tag":123,"props":451,"children":452},{"style":145},[453],{"type":45,"value":454},".data.spans[] | select(.parent_span_id == null)",{"type":39,"tag":123,"props":456,"children":457},{"style":171},[458],{"type":45,"value":217},{"type":39,"tag":123,"props":460,"children":461},{"style":145},[462],{"type":45,"value":394},{"type":39,"tag":55,"props":464,"children":465},{},[466,468,474],{"type":45,"value":467},"Examine its ",{"type":39,"tag":66,"props":469,"children":471},{"className":470},[],[472],{"type":45,"value":473},"attributes",{"type":45,"value":475}," dict to identify which keys hold the user input and system output. These could be:",{"type":39,"tag":477,"props":478,"children":479},"ul",{},[480,507,533],{"type":39,"tag":481,"props":482,"children":483},"li",{},[484,489,491,497,499,505],{"type":39,"tag":86,"props":485,"children":486},{},[487],{"type":45,"value":488},"MLflow standard attributes",{"type":45,"value":490},": ",{"type":39,"tag":66,"props":492,"children":494},{"className":493},[],[495],{"type":45,"value":496},"mlflow.spanInputs",{"type":45,"value":498}," and ",{"type":39,"tag":66,"props":500,"children":502},{"className":501},[],[503],{"type":45,"value":504},"mlflow.spanOutputs",{"type":45,"value":506}," (set by the MLflow Python client)",{"type":39,"tag":481,"props":508,"children":509},{},[510,515,517,523,525,531],{"type":39,"tag":86,"props":511,"children":512},{},[513],{"type":45,"value":514},"Custom attributes",{"type":45,"value":516},": Application-specific keys set via ",{"type":39,"tag":66,"props":518,"children":520},{"className":519},[],[521],{"type":45,"value":522},"@mlflow.trace",{"type":45,"value":524}," or ",{"type":39,"tag":66,"props":526,"children":528},{"className":527},[],[529],{"type":45,"value":530},"mlflow.start_span()",{"type":45,"value":532}," with custom attribute logging",{"type":39,"tag":481,"props":534,"children":535},{},[536,541],{"type":39,"tag":86,"props":537,"children":538},{},[539],{"type":45,"value":540},"Third-party OTel attributes",{"type":45,"value":542},": Keys following GenAI Semantic Conventions, OpenInference, or other instrumentation conventions",{"type":39,"tag":55,"props":544,"children":545},{},[546,548,554,556,562],{"type":45,"value":547},"The structure of these values also varies by application (e.g., a ",{"type":39,"tag":66,"props":549,"children":551},{"className":550},[],[552],{"type":45,"value":553},"query",{"type":45,"value":555}," string, a ",{"type":39,"tag":66,"props":557,"children":559},{"className":558},[],[560],{"type":45,"value":561},"messages",{"type":45,"value":563}," array, a dict with multiple fields). Inspect the actual attribute values to understand the format.",{"type":39,"tag":55,"props":565,"children":566},{},[567,572],{"type":39,"tag":86,"props":568,"children":569},{},[570],{"type":45,"value":571},"If the root span has empty or missing inputs\u002Foutputs",{"type":45,"value":573},", it may be a wrapper span (e.g., an orchestrator or middleware) that doesn't directly carry the chat turn data. In that case, look at its immediate children — find the closest span to the top of the hierarchy that has meaningful inputs and outputs corresponding to a chat turn:",{"type":39,"tag":55,"props":575,"children":576},{},[577,579,584,586,591],{"type":45,"value":578},"The following example assumes the trace comes from the MLflow Python client (which stores inputs\u002Foutputs in ",{"type":39,"tag":66,"props":580,"children":582},{"className":581},[],[583],{"type":45,"value":496},{"type":45,"value":585},"\u002F",{"type":39,"tag":66,"props":587,"children":589},{"className":588},[],[590],{"type":45,"value":504},{"type":45,"value":592},") and that the relevant span is a direct child of root. In practice, the relevant span may be deeper in the hierarchy, and traces from other clients may use different attribute keys — explore the span tree as needed:",{"type":39,"tag":112,"props":594,"children":596},{"className":114,"code":595,"language":116,"meta":117,"style":117},"# Get the root span's ID\nROOT_ID=$(jq -r '.data.spans[] | select(.parent_span_id == null) | .span_id' \u002Ftmp\u002Ftrace_detail.json)\n\n# List immediate children of the root span with their inputs\u002Foutputs\njq --arg root \"$ROOT_ID\" '.data.spans[] | select(.parent_span_id == $root) | {name: .name, inputs: .attributes[\"mlflow.spanInputs\"], outputs: .attributes[\"mlflow.spanOutputs\"]}' \u002Ftmp\u002Ftrace_detail.json\n",[597],{"type":39,"tag":66,"props":598,"children":599},{"__ignoreMap":117},[600,608,653,660,668],{"type":39,"tag":123,"props":601,"children":602},{"class":125,"line":126},[603],{"type":39,"tag":123,"props":604,"children":605},{"style":130},[606],{"type":45,"value":607},"# Get the root span's ID\n",{"type":39,"tag":123,"props":609,"children":610},{"class":125,"line":136},[611,616,621,625,630,634,639,643,648],{"type":39,"tag":123,"props":612,"children":613},{"style":156},[614],{"type":45,"value":615},"ROOT_ID",{"type":39,"tag":123,"props":617,"children":618},{"style":171},[619],{"type":45,"value":620},"=$(",{"type":39,"tag":123,"props":622,"children":623},{"style":140},[624],{"type":45,"value":445},{"type":39,"tag":123,"props":626,"children":627},{"style":145},[628],{"type":45,"value":629}," -r",{"type":39,"tag":123,"props":631,"children":632},{"style":171},[633],{"type":45,"value":207},{"type":39,"tag":123,"props":635,"children":636},{"style":145},[637],{"type":45,"value":638},".data.spans[] | select(.parent_span_id == null) | .span_id",{"type":39,"tag":123,"props":640,"children":641},{"style":171},[642],{"type":45,"value":217},{"type":39,"tag":123,"props":644,"children":645},{"style":145},[646],{"type":45,"value":647}," \u002Ftmp\u002Ftrace_detail.json",{"type":39,"tag":123,"props":649,"children":650},{"style":171},[651],{"type":45,"value":652},")\n",{"type":39,"tag":123,"props":654,"children":655},{"class":125,"line":162},[656],{"type":39,"tag":123,"props":657,"children":658},{"emptyLinePlaceholder":325},[659],{"type":45,"value":328},{"type":39,"tag":123,"props":661,"children":662},{"class":125,"line":196},[663],{"type":39,"tag":123,"props":664,"children":665},{"style":130},[666],{"type":45,"value":667},"# List immediate children of the root span with their inputs\u002Foutputs\n",{"type":39,"tag":123,"props":669,"children":670},{"class":125,"line":224},[671,675,680,685,689,694,698,702,707,711],{"type":39,"tag":123,"props":672,"children":673},{"style":140},[674],{"type":45,"value":445},{"type":39,"tag":123,"props":676,"children":677},{"style":145},[678],{"type":45,"value":679}," --arg",{"type":39,"tag":123,"props":681,"children":682},{"style":145},[683],{"type":45,"value":684}," root",{"type":39,"tag":123,"props":686,"children":687},{"style":171},[688],{"type":45,"value":235},{"type":39,"tag":123,"props":690,"children":691},{"style":156},[692],{"type":45,"value":693},"$ROOT_ID",{"type":39,"tag":123,"props":695,"children":696},{"style":171},[697],{"type":45,"value":245},{"type":39,"tag":123,"props":699,"children":700},{"style":171},[701],{"type":45,"value":207},{"type":39,"tag":123,"props":703,"children":704},{"style":145},[705],{"type":45,"value":706},".data.spans[] | select(.parent_span_id == $root) | {name: .name, inputs: .attributes[\"mlflow.spanInputs\"], outputs: .attributes[\"mlflow.spanOutputs\"]}",{"type":39,"tag":123,"props":708,"children":709},{"style":171},[710],{"type":45,"value":217},{"type":39,"tag":123,"props":712,"children":713},{"style":145},[714],{"type":45,"value":394},{"type":39,"tag":55,"props":716,"children":717},{},[718,720,725],{"type":45,"value":719},"Also check the first trace's assessments. ",{"type":39,"tag":86,"props":721,"children":722},{},[723],{"type":45,"value":724},"Session-level assessments are attached to the first trace in the session",{"type":45,"value":726}," — these evaluate the session as a whole (e.g., overall conversation quality, multi-turn coherence) and can indicate the presence of issues somewhere across the entire session, not just the first turn. The first trace may also have per-turn assessments for that specific turn.",{"type":39,"tag":55,"props":728,"children":729},{},[730,732,738,740,745,747,753],{"type":45,"value":731},"Both types appear in ",{"type":39,"tag":66,"props":733,"children":735},{"className":734},[],[736],{"type":45,"value":737},".info.assessments",{"type":45,"value":739},". Session-level assessments are identified by the presence of ",{"type":39,"tag":66,"props":741,"children":743},{"className":742},[],[744],{"type":45,"value":71},{"type":45,"value":746}," in their ",{"type":39,"tag":66,"props":748,"children":750},{"className":749},[],[751],{"type":45,"value":752},"metadata",{"type":45,"value":754}," field:",{"type":39,"tag":112,"props":756,"children":758},{"className":114,"code":757,"language":116,"meta":117,"style":117},"# Show session-level assessments (exclude scorer errors)\njq '[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"]) | {name: .assessment_name, value: .feedback.value}]' \u002Ftmp\u002Ftrace_detail.json\n\n# Show per-turn assessments (exclude scorer errors)\njq '[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"] == null) | {name: .assessment_name, value: .feedback.value}]' \u002Ftmp\u002Ftrace_detail.json\n",[759],{"type":39,"tag":66,"props":760,"children":761},{"__ignoreMap":117},[762,770,794,801,809],{"type":39,"tag":123,"props":763,"children":764},{"class":125,"line":126},[765],{"type":39,"tag":123,"props":766,"children":767},{"style":130},[768],{"type":45,"value":769},"# Show session-level assessments (exclude scorer errors)\n",{"type":39,"tag":123,"props":771,"children":772},{"class":125,"line":136},[773,777,781,786,790],{"type":39,"tag":123,"props":774,"children":775},{"style":140},[776],{"type":45,"value":445},{"type":39,"tag":123,"props":778,"children":779},{"style":171},[780],{"type":45,"value":207},{"type":39,"tag":123,"props":782,"children":783},{"style":145},[784],{"type":45,"value":785},"[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"]) | {name: .assessment_name, value: .feedback.value}]",{"type":39,"tag":123,"props":787,"children":788},{"style":171},[789],{"type":45,"value":217},{"type":39,"tag":123,"props":791,"children":792},{"style":145},[793],{"type":45,"value":394},{"type":39,"tag":123,"props":795,"children":796},{"class":125,"line":162},[797],{"type":39,"tag":123,"props":798,"children":799},{"emptyLinePlaceholder":325},[800],{"type":45,"value":328},{"type":39,"tag":123,"props":802,"children":803},{"class":125,"line":196},[804],{"type":39,"tag":123,"props":805,"children":806},{"style":130},[807],{"type":45,"value":808},"# Show per-turn assessments (exclude scorer errors)\n",{"type":39,"tag":123,"props":810,"children":811},{"class":125,"line":224},[812,816,820,825,829],{"type":39,"tag":123,"props":813,"children":814},{"style":140},[815],{"type":45,"value":445},{"type":39,"tag":123,"props":817,"children":818},{"style":171},[819],{"type":45,"value":207},{"type":39,"tag":123,"props":821,"children":822},{"style":145},[823],{"type":45,"value":824},"[.info.assessments[] | select(.feedback.error == null) | select(.metadata[\"mlflow.trace.session\"] == null) | {name: .assessment_name, value: .feedback.value}]",{"type":39,"tag":123,"props":826,"children":827},{"style":171},[828],{"type":45,"value":217},{"type":39,"tag":123,"props":830,"children":831},{"style":145},[832],{"type":45,"value":394},{"type":39,"tag":55,"props":834,"children":835},{},[836,841,843,849],{"type":39,"tag":86,"props":837,"children":838},{},[839],{"type":45,"value":840},"Assessment errors are not trace errors.",{"type":45,"value":842}," If an assessment has a ",{"type":39,"tag":66,"props":844,"children":846},{"className":845},[],[847],{"type":45,"value":848},"feedback.error",{"type":45,"value":850}," field, it means the scorer or judge failed — not that the trace itself has a problem. Exclude these when using assessments to identify trace issues.",{"type":39,"tag":55,"props":852,"children":853},{},[854,859,861,867,869,875,877,883,885,891,893,899,901,906,908,914],{"type":39,"tag":86,"props":855,"children":856},{},[857],{"type":45,"value":858},"Always consult the rationale when interpreting assessment values.",{"type":45,"value":860}," The ",{"type":39,"tag":66,"props":862,"children":864},{"className":863},[],[865],{"type":45,"value":866},"value",{"type":45,"value":868}," alone can be misleading — for example, a ",{"type":39,"tag":66,"props":870,"children":872},{"className":871},[],[873],{"type":45,"value":874},"user_frustration",{"type":45,"value":876}," assessment with ",{"type":39,"tag":66,"props":878,"children":880},{"className":879},[],[881],{"type":45,"value":882},"value: \"no\"",{"type":45,"value":884}," could mean \"no frustration detected\" or \"the frustration check did not pass\" (i.e., frustration ",{"type":39,"tag":886,"props":887,"children":888},"em",{},[889],{"type":45,"value":890},"is",{"type":45,"value":892}," present), depending on how the scorer was configured. The ",{"type":39,"tag":66,"props":894,"children":896},{"className":895},[],[897],{"type":45,"value":898},".rationale",{"type":45,"value":900}," field (a top-level assessment field, ",{"type":39,"tag":86,"props":902,"children":903},{},[904],{"type":45,"value":905},"not",{"type":45,"value":907}," nested under ",{"type":39,"tag":66,"props":909,"children":911},{"className":910},[],[912],{"type":45,"value":913},".feedback",{"type":45,"value":915},") explains what the value means in context. Include rationale when extracting assessments:",{"type":39,"tag":112,"props":917,"children":919},{"className":114,"code":918,"language":116,"meta":117,"style":117},"jq '[.info.assessments[] | select(.feedback.error == null) | {name: .assessment_name, value: .feedback.value, rationale: .rationale}]' \u002Ftmp\u002Ftrace_detail.json\n",[920],{"type":39,"tag":66,"props":921,"children":922},{"__ignoreMap":117},[923],{"type":39,"tag":123,"props":924,"children":925},{"class":125,"line":126},[926,930,934,939,943],{"type":39,"tag":123,"props":927,"children":928},{"style":140},[929],{"type":45,"value":445},{"type":39,"tag":123,"props":931,"children":932},{"style":171},[933],{"type":45,"value":207},{"type":39,"tag":123,"props":935,"children":936},{"style":145},[937],{"type":45,"value":938},"[.info.assessments[] | select(.feedback.error == null) | {name: .assessment_name, value: .feedback.value, rationale: .rationale}]",{"type":39,"tag":123,"props":940,"children":941},{"style":171},[942],{"type":45,"value":217},{"type":39,"tag":123,"props":944,"children":945},{"style":145},[946],{"type":45,"value":394},{"type":39,"tag":55,"props":948,"children":949},{},[950,955,957,962,964,971],{"type":39,"tag":86,"props":951,"children":952},{},[953],{"type":45,"value":954},"Step 2: Extract across all session traces.",{"type":45,"value":956}," Once you know which attribute keys hold inputs and outputs, search for all traces in the session using ",{"type":39,"tag":66,"props":958,"children":960},{"className":959},[],[961],{"type":45,"value":98},{"type":45,"value":963}," to pull those fields along with assessments (see ",{"type":39,"tag":965,"props":966,"children":968},"a",{"href":967},"#handling-cli-output",[969],{"type":45,"value":970},"Handling CLI Output",{"type":45,"value":972}," for why output is written to a file):",{"type":39,"tag":112,"props":974,"children":976},{"className":114,"code":975,"language":116,"meta":117,"style":117},"mlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  --filter-string 'metadata.`mlflow.trace.session` = \"\u003CSESSION_ID>\"' \\\n  --order-by \"timestamp_ms ASC\" \\\n  --extract-fields 'info.trace_id,info.state,info.request_time,info.assessments,info.trace_metadata.`mlflow.traceInputs`,info.trace_metadata.`mlflow.traceOutputs`' \\\n  --output json \\\n  --max-results 100 > \u002Ftmp\u002Fsession_traces.json\n",[977],{"type":39,"tag":66,"props":978,"children":979},{"__ignoreMap":117},[980,999,1026,1049,1072,1096,1111],{"type":39,"tag":123,"props":981,"children":982},{"class":125,"line":126},[983,987,991,995],{"type":39,"tag":123,"props":984,"children":985},{"style":140},[986],{"type":45,"value":8},{"type":39,"tag":123,"props":988,"children":989},{"style":145},[990],{"type":45,"value":148},{"type":39,"tag":123,"props":992,"children":993},{"style":145},[994],{"type":45,"value":153},{"type":39,"tag":123,"props":996,"children":997},{"style":156},[998],{"type":45,"value":159},{"type":39,"tag":123,"props":1000,"children":1001},{"class":125,"line":136},[1002,1006,1010,1014,1018,1022],{"type":39,"tag":123,"props":1003,"children":1004},{"style":145},[1005],{"type":45,"value":168},{"type":39,"tag":123,"props":1007,"children":1008},{"style":171},[1009],{"type":45,"value":174},{"type":39,"tag":123,"props":1011,"children":1012},{"style":145},[1013],{"type":45,"value":179},{"type":39,"tag":123,"props":1015,"children":1016},{"style":156},[1017],{"type":45,"value":184},{"type":39,"tag":123,"props":1019,"children":1020},{"style":171},[1021],{"type":45,"value":189},{"type":39,"tag":123,"props":1023,"children":1024},{"style":156},[1025],{"type":45,"value":159},{"type":39,"tag":123,"props":1027,"children":1028},{"class":125,"line":162},[1029,1033,1037,1041,1045],{"type":39,"tag":123,"props":1030,"children":1031},{"style":145},[1032],{"type":45,"value":202},{"type":39,"tag":123,"props":1034,"children":1035},{"style":171},[1036],{"type":45,"value":207},{"type":39,"tag":123,"props":1038,"children":1039},{"style":145},[1040],{"type":45,"value":212},{"type":39,"tag":123,"props":1042,"children":1043},{"style":171},[1044],{"type":45,"value":217},{"type":39,"tag":123,"props":1046,"children":1047},{"style":156},[1048],{"type":45,"value":159},{"type":39,"tag":123,"props":1050,"children":1051},{"class":125,"line":196},[1052,1056,1060,1064,1068],{"type":39,"tag":123,"props":1053,"children":1054},{"style":145},[1055],{"type":45,"value":230},{"type":39,"tag":123,"props":1057,"children":1058},{"style":171},[1059],{"type":45,"value":235},{"type":39,"tag":123,"props":1061,"children":1062},{"style":145},[1063],{"type":45,"value":240},{"type":39,"tag":123,"props":1065,"children":1066},{"style":171},[1067],{"type":45,"value":245},{"type":39,"tag":123,"props":1069,"children":1070},{"style":156},[1071],{"type":45,"value":159},{"type":39,"tag":123,"props":1073,"children":1074},{"class":125,"line":224},[1075,1079,1083,1088,1092],{"type":39,"tag":123,"props":1076,"children":1077},{"style":145},[1078],{"type":45,"value":258},{"type":39,"tag":123,"props":1080,"children":1081},{"style":171},[1082],{"type":45,"value":207},{"type":39,"tag":123,"props":1084,"children":1085},{"style":145},[1086],{"type":45,"value":1087},"info.trace_id,info.state,info.request_time,info.assessments,info.trace_metadata.`mlflow.traceInputs`,info.trace_metadata.`mlflow.traceOutputs`",{"type":39,"tag":123,"props":1089,"children":1090},{"style":171},[1091],{"type":45,"value":217},{"type":39,"tag":123,"props":1093,"children":1094},{"style":156},[1095],{"type":45,"value":159},{"type":39,"tag":123,"props":1097,"children":1098},{"class":125,"line":252},[1099,1103,1107],{"type":39,"tag":123,"props":1100,"children":1101},{"style":145},[1102],{"type":45,"value":284},{"type":39,"tag":123,"props":1104,"children":1105},{"style":145},[1106],{"type":45,"value":289},{"type":39,"tag":123,"props":1108,"children":1109},{"style":156},[1110],{"type":45,"value":159},{"type":39,"tag":123,"props":1112,"children":1113},{"class":125,"line":278},[1114,1118,1123,1127],{"type":39,"tag":123,"props":1115,"children":1116},{"style":145},[1117],{"type":45,"value":302},{"type":39,"tag":123,"props":1119,"children":1120},{"style":305},[1121],{"type":45,"value":1122}," 100",{"type":39,"tag":123,"props":1124,"children":1125},{"style":171},[1126],{"type":45,"value":313},{"type":39,"tag":123,"props":1128,"children":1129},{"style":145},[1130],{"type":45,"value":1131}," \u002Ftmp\u002Fsession_traces.json\n",{"type":39,"tag":55,"props":1133,"children":1134},{},[1135,1137,1142,1144,1150,1151,1157],{"type":45,"value":1136},"Then use bash commands (e.g., ",{"type":39,"tag":66,"props":1138,"children":1140},{"className":1139},[],[1141],{"type":45,"value":445},{"type":45,"value":1143},", ",{"type":39,"tag":66,"props":1145,"children":1147},{"className":1146},[],[1148],{"type":45,"value":1149},"wc",{"type":45,"value":1143},{"type":39,"tag":66,"props":1152,"children":1154},{"className":1153},[],[1155],{"type":45,"value":1156},"head",{"type":45,"value":1158},") on the file to analyze it.",{"type":39,"tag":55,"props":1160,"children":1161},{},[1162,1164,1169,1171,1177,1178,1184],{"type":45,"value":1163},"The ",{"type":39,"tag":66,"props":1165,"children":1167},{"className":1166},[],[1168],{"type":45,"value":98},{"type":45,"value":1170}," example above uses ",{"type":39,"tag":66,"props":1172,"children":1174},{"className":1173},[],[1175],{"type":45,"value":1176},"mlflow.traceInputs",{"type":45,"value":585},{"type":39,"tag":66,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":45,"value":1183},"mlflow.traceOutputs",{"type":45,"value":1185}," from trace metadata — adjust the field paths based on what you discovered in step 1.",{"type":39,"tag":55,"props":1187,"children":1188},{},[1189],{"type":45,"value":1190},"Assessments contain quality judgments (e.g., correctness, relevance) that can pinpoint which turns had issues without needing to read every trace in detail. To identify which turns have assessment signals (excluding scorer errors):",{"type":39,"tag":112,"props":1192,"children":1194},{"className":114,"code":1193,"language":116,"meta":117,"style":117},"# List turns with their valid assessments (scorer errors filtered out)\njq '.traces[] | {\n  trace_id: .info.trace_id,\n  time: .info.request_time,\n  state: .info.state,\n  assessments: [.info.assessments[]? | select(.feedback.error == null) | {\n    name: .assessment_name,\n    value: .feedback.value\n  }]\n}' \u002Ftmp\u002Fsession_traces.json\n",[1195],{"type":39,"tag":66,"props":1196,"children":1197},{"__ignoreMap":117},[1198,1206,1222,1230,1238,1246,1254,1262,1270,1278],{"type":39,"tag":123,"props":1199,"children":1200},{"class":125,"line":126},[1201],{"type":39,"tag":123,"props":1202,"children":1203},{"style":130},[1204],{"type":45,"value":1205},"# List turns with their valid assessments (scorer errors filtered out)\n",{"type":39,"tag":123,"props":1207,"children":1208},{"class":125,"line":136},[1209,1213,1217],{"type":39,"tag":123,"props":1210,"children":1211},{"style":140},[1212],{"type":45,"value":445},{"type":39,"tag":123,"props":1214,"children":1215},{"style":171},[1216],{"type":45,"value":207},{"type":39,"tag":123,"props":1218,"children":1219},{"style":145},[1220],{"type":45,"value":1221},".traces[] | {\n",{"type":39,"tag":123,"props":1223,"children":1224},{"class":125,"line":162},[1225],{"type":39,"tag":123,"props":1226,"children":1227},{"style":145},[1228],{"type":45,"value":1229},"  trace_id: .info.trace_id,\n",{"type":39,"tag":123,"props":1231,"children":1232},{"class":125,"line":196},[1233],{"type":39,"tag":123,"props":1234,"children":1235},{"style":145},[1236],{"type":45,"value":1237},"  time: .info.request_time,\n",{"type":39,"tag":123,"props":1239,"children":1240},{"class":125,"line":224},[1241],{"type":39,"tag":123,"props":1242,"children":1243},{"style":145},[1244],{"type":45,"value":1245},"  state: .info.state,\n",{"type":39,"tag":123,"props":1247,"children":1248},{"class":125,"line":252},[1249],{"type":39,"tag":123,"props":1250,"children":1251},{"style":145},[1252],{"type":45,"value":1253},"  assessments: [.info.assessments[]? | select(.feedback.error == null) | {\n",{"type":39,"tag":123,"props":1255,"children":1256},{"class":125,"line":278},[1257],{"type":39,"tag":123,"props":1258,"children":1259},{"style":145},[1260],{"type":45,"value":1261},"    name: .assessment_name,\n",{"type":39,"tag":123,"props":1263,"children":1264},{"class":125,"line":296},[1265],{"type":39,"tag":123,"props":1266,"children":1267},{"style":145},[1268],{"type":45,"value":1269},"    value: .feedback.value\n",{"type":39,"tag":123,"props":1271,"children":1272},{"class":125,"line":321},[1273],{"type":39,"tag":123,"props":1274,"children":1275},{"style":145},[1276],{"type":45,"value":1277},"  }]\n",{"type":39,"tag":123,"props":1279,"children":1280},{"class":125,"line":331},[1281,1286,1290],{"type":39,"tag":123,"props":1282,"children":1283},{"style":145},[1284],{"type":45,"value":1285},"}",{"type":39,"tag":123,"props":1287,"children":1288},{"style":171},[1289],{"type":45,"value":217},{"type":39,"tag":123,"props":1291,"children":1292},{"style":145},[1293],{"type":45,"value":1131},{"type":39,"tag":55,"props":1295,"children":1296},{},[1297],{"type":39,"tag":86,"props":1298,"children":1299},{},[1300],{"type":45,"value":1301},"CLI syntax notes:",{"type":39,"tag":477,"props":1303,"children":1304},{},[1305,1329,1347,1387],{"type":39,"tag":481,"props":1306,"children":1307},{},[1308,1319,1321,1327],{"type":39,"tag":86,"props":1309,"children":1310},{},[1311,1317],{"type":39,"tag":66,"props":1312,"children":1314},{"className":1313},[],[1315],{"type":45,"value":1316},"--experiment-id",{"type":45,"value":1318}," is required",{"type":45,"value":1320}," for all ",{"type":39,"tag":66,"props":1322,"children":1324},{"className":1323},[],[1325],{"type":45,"value":1326},"mlflow traces search",{"type":45,"value":1328}," commands. The command will fail without it.",{"type":39,"tag":481,"props":1330,"children":1331},{},[1332,1334,1339,1341],{"type":45,"value":1333},"Metadata keys containing dots ",{"type":39,"tag":86,"props":1335,"children":1336},{},[1337],{"type":45,"value":1338},"must",{"type":45,"value":1340}," be escaped with backticks in filter strings and extract-fields: ",{"type":39,"tag":66,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":45,"value":1346},"metadata.`mlflow.trace.session`",{"type":39,"tag":481,"props":1348,"children":1349},{},[1350,1355,1357,1362,1364,1370,1372,1377,1379,1385],{"type":39,"tag":86,"props":1351,"children":1352},{},[1353],{"type":45,"value":1354},"Shell quoting",{"type":45,"value":1356},": Backticks inside ",{"type":39,"tag":86,"props":1358,"children":1359},{},[1360],{"type":45,"value":1361},"double quotes",{"type":45,"value":1363}," are interpreted by bash as command substitution (e.g., bash will try to run ",{"type":39,"tag":66,"props":1365,"children":1367},{"className":1366},[],[1368],{"type":45,"value":1369},"`mlflow.trace.session`",{"type":45,"value":1371}," as a command). Always use ",{"type":39,"tag":86,"props":1373,"children":1374},{},[1375],{"type":45,"value":1376},"single quotes",{"type":45,"value":1378}," for the outer string when the value contains backticks. For example: ",{"type":39,"tag":66,"props":1380,"children":1382},{"className":1381},[],[1383],{"type":45,"value":1384},"--filter-string 'metadata.\\",{"type":45,"value":1386},"mlflow.trace.session` = \"value\"'`",{"type":39,"tag":481,"props":1388,"children":1389},{},[1390,1396],{"type":39,"tag":66,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":45,"value":1395},"--max-results",{"type":45,"value":1397}," defaults to 100, which is sufficient for most sessions. Increase up to 500 (the maximum) for longer conversations. If 500 results are returned, use pagination to retrieve the rest.",{"type":39,"tag":48,"props":1399,"children":1401},{"id":1400},"handling-cli-output",[1402],{"type":45,"value":970},{"type":39,"tag":55,"props":1404,"children":1405},{},[1406],{"type":45,"value":1407},"MLflow trace output can be large, and Claude Code's Bash tool has a ~30KB output limit for piped commands. When output exceeds this threshold, it gets saved to a file instead of being piped, causing silent failures.",{"type":39,"tag":55,"props":1409,"children":1410},{},[1411],{"type":39,"tag":86,"props":1412,"children":1413},{},[1414],{"type":45,"value":1415},"Safe approach (always works):",{"type":39,"tag":112,"props":1417,"children":1419},{"className":114,"code":1418,"language":116,"meta":117,"style":117},"# Step 1: Save to file\nmlflow traces search \\\n  --experiment-id \u003CEXPERIMENT_ID> \\\n  [...] \\\n  --output json > \u002Ftmp\u002Foutput.json\n\n# Step 2: Process the file\ncat \u002Ftmp\u002Foutput.json | jq '.traces[0].info.trace_id'\nhead -50 \u002Ftmp\u002Foutput.json\nwc -l \u002Ftmp\u002Foutput.json\n",[1420],{"type":39,"tag":66,"props":1421,"children":1422},{"__ignoreMap":117},[1423,1431,1450,1477,1485,1505,1512,1520,1557,1573],{"type":39,"tag":123,"props":1424,"children":1425},{"class":125,"line":126},[1426],{"type":39,"tag":123,"props":1427,"children":1428},{"style":130},[1429],{"type":45,"value":1430},"# Step 1: Save to file\n",{"type":39,"tag":123,"props":1432,"children":1433},{"class":125,"line":136},[1434,1438,1442,1446],{"type":39,"tag":123,"props":1435,"children":1436},{"style":140},[1437],{"type":45,"value":8},{"type":39,"tag":123,"props":1439,"children":1440},{"style":145},[1441],{"type":45,"value":148},{"type":39,"tag":123,"props":1443,"children":1444},{"style":145},[1445],{"type":45,"value":153},{"type":39,"tag":123,"props":1447,"children":1448},{"style":156},[1449],{"type":45,"value":159},{"type":39,"tag":123,"props":1451,"children":1452},{"class":125,"line":162},[1453,1457,1461,1465,1469,1473],{"type":39,"tag":123,"props":1454,"children":1455},{"style":145},[1456],{"type":45,"value":168},{"type":39,"tag":123,"props":1458,"children":1459},{"style":171},[1460],{"type":45,"value":174},{"type":39,"tag":123,"props":1462,"children":1463},{"style":145},[1464],{"type":45,"value":179},{"type":39,"tag":123,"props":1466,"children":1467},{"style":156},[1468],{"type":45,"value":184},{"type":39,"tag":123,"props":1470,"children":1471},{"style":171},[1472],{"type":45,"value":189},{"type":39,"tag":123,"props":1474,"children":1475},{"style":156},[1476],{"type":45,"value":159},{"type":39,"tag":123,"props":1478,"children":1479},{"class":125,"line":196},[1480],{"type":39,"tag":123,"props":1481,"children":1482},{"style":156},[1483],{"type":45,"value":1484},"  [...] \\\n",{"type":39,"tag":123,"props":1486,"children":1487},{"class":125,"line":224},[1488,1492,1496,1500],{"type":39,"tag":123,"props":1489,"children":1490},{"style":140},[1491],{"type":45,"value":284},{"type":39,"tag":123,"props":1493,"children":1494},{"style":145},[1495],{"type":45,"value":289},{"type":39,"tag":123,"props":1497,"children":1498},{"style":171},[1499],{"type":45,"value":313},{"type":39,"tag":123,"props":1501,"children":1502},{"style":145},[1503],{"type":45,"value":1504}," \u002Ftmp\u002Foutput.json\n",{"type":39,"tag":123,"props":1506,"children":1507},{"class":125,"line":252},[1508],{"type":39,"tag":123,"props":1509,"children":1510},{"emptyLinePlaceholder":325},[1511],{"type":45,"value":328},{"type":39,"tag":123,"props":1513,"children":1514},{"class":125,"line":278},[1515],{"type":39,"tag":123,"props":1516,"children":1517},{"style":130},[1518],{"type":45,"value":1519},"# Step 2: Process the file\n",{"type":39,"tag":123,"props":1521,"children":1522},{"class":125,"line":296},[1523,1528,1533,1538,1543,1547,1552],{"type":39,"tag":123,"props":1524,"children":1525},{"style":140},[1526],{"type":45,"value":1527},"cat",{"type":39,"tag":123,"props":1529,"children":1530},{"style":145},[1531],{"type":45,"value":1532}," \u002Ftmp\u002Foutput.json",{"type":39,"tag":123,"props":1534,"children":1535},{"style":171},[1536],{"type":45,"value":1537}," |",{"type":39,"tag":123,"props":1539,"children":1540},{"style":140},[1541],{"type":45,"value":1542}," jq",{"type":39,"tag":123,"props":1544,"children":1545},{"style":171},[1546],{"type":45,"value":207},{"type":39,"tag":123,"props":1548,"children":1549},{"style":145},[1550],{"type":45,"value":1551},".traces[0].info.trace_id",{"type":39,"tag":123,"props":1553,"children":1554},{"style":171},[1555],{"type":45,"value":1556},"'\n",{"type":39,"tag":123,"props":1558,"children":1559},{"class":125,"line":321},[1560,1564,1569],{"type":39,"tag":123,"props":1561,"children":1562},{"style":140},[1563],{"type":45,"value":1156},{"type":39,"tag":123,"props":1565,"children":1566},{"style":145},[1567],{"type":45,"value":1568}," -50",{"type":39,"tag":123,"props":1570,"children":1571},{"style":145},[1572],{"type":45,"value":1504},{"type":39,"tag":123,"props":1574,"children":1575},{"class":125,"line":331},[1576,1580,1585],{"type":39,"tag":123,"props":1577,"children":1578},{"style":140},[1579],{"type":45,"value":1149},{"type":39,"tag":123,"props":1581,"children":1582},{"style":145},[1583],{"type":45,"value":1584}," -l",{"type":39,"tag":123,"props":1586,"children":1587},{"style":145},[1588],{"type":45,"value":1504},{"type":39,"tag":55,"props":1590,"children":1591},{},[1592,1597,1599,1605],{"type":39,"tag":86,"props":1593,"children":1594},{},[1595],{"type":45,"value":1596},"Never pipe MLflow CLI output directly",{"type":45,"value":1598}," (e.g., ",{"type":39,"tag":66,"props":1600,"children":1602},{"className":1601},[],[1603],{"type":45,"value":1604},"mlflow traces search ... | jq '.'",{"type":45,"value":1606},"). This can silently produce no output. Always redirect to a file first, then run commands on the file.",{"type":39,"tag":55,"props":1608,"children":1609},{},[1610],{"type":45,"value":1611},"To inspect a specific turn in detail (e.g., after identifying a problematic turn), fetch its full trace:",{"type":39,"tag":112,"props":1613,"children":1615},{"className":114,"code":1614,"language":116,"meta":117,"style":117},"mlflow traces get --trace-id \u003CTRACE_ID> > \u002Ftmp\u002Fturn_detail.json\n",[1616],{"type":39,"tag":66,"props":1617,"children":1618},{"__ignoreMap":117},[1619],{"type":39,"tag":123,"props":1620,"children":1621},{"class":125,"line":126},[1622,1626,1630,1634,1639,1643,1648,1652,1656,1660],{"type":39,"tag":123,"props":1623,"children":1624},{"style":140},[1625],{"type":45,"value":8},{"type":39,"tag":123,"props":1627,"children":1628},{"style":145},[1629],{"type":45,"value":148},{"type":39,"tag":123,"props":1631,"children":1632},{"style":145},[1633],{"type":45,"value":354},{"type":39,"tag":123,"props":1635,"children":1636},{"style":145},[1637],{"type":45,"value":1638}," --trace-id",{"type":39,"tag":123,"props":1640,"children":1641},{"style":171},[1642],{"type":45,"value":174},{"type":39,"tag":123,"props":1644,"children":1645},{"style":145},[1646],{"type":45,"value":1647},"TRACE_I",{"type":39,"tag":123,"props":1649,"children":1650},{"style":156},[1651],{"type":45,"value":184},{"type":39,"tag":123,"props":1653,"children":1654},{"style":171},[1655],{"type":45,"value":189},{"type":39,"tag":123,"props":1657,"children":1658},{"style":171},[1659],{"type":45,"value":313},{"type":39,"tag":123,"props":1661,"children":1662},{"style":145},[1663],{"type":45,"value":1664}," \u002Ftmp\u002Fturn_detail.json\n",{"type":39,"tag":48,"props":1666,"children":1668},{"id":1667},"codebase-correlation",[1669],{"type":45,"value":1670},"Codebase Correlation",{"type":39,"tag":477,"props":1672,"children":1673},{},[1674,1691,1701],{"type":39,"tag":481,"props":1675,"children":1676},{},[1677,1682,1684,1689],{"type":39,"tag":86,"props":1678,"children":1679},{},[1680],{"type":45,"value":1681},"Session ID assignment",{"type":45,"value":1683},": Search the codebase for where ",{"type":39,"tag":66,"props":1685,"children":1687},{"className":1686},[],[1688],{"type":45,"value":71},{"type":45,"value":1690}," is set to understand how sessions are created — per user login, per browser tab, per explicit \"new conversation\" action, etc.",{"type":39,"tag":481,"props":1692,"children":1693},{},[1694,1699],{"type":39,"tag":86,"props":1695,"children":1696},{},[1697],{"type":45,"value":1698},"Context window management",{"type":45,"value":1700},": Look for how the application constructs the message history passed to the LLM at each turn. Common patterns include sliding window (last N messages), summarization of older turns, or full history. This implementation determines what context the model sees and is a frequent source of multi-turn failures.",{"type":39,"tag":481,"props":1702,"children":1703},{},[1704,1709],{"type":39,"tag":86,"props":1705,"children":1706},{},[1707],{"type":45,"value":1708},"Memory and state",{"type":45,"value":1710},": Some applications maintain state across turns beyond message history (e.g., extracted entities, user preferences, accumulated tool results). Search for how this state is stored and passed between turns.",{"type":39,"tag":48,"props":1712,"children":1714},{"id":1713},"reference-scripts",[1715],{"type":45,"value":1716},"Reference Scripts",{"type":39,"tag":55,"props":1718,"children":1719},{},[1720,1721,1727],{"type":45,"value":1163},{"type":39,"tag":66,"props":1722,"children":1724},{"className":1723},[],[1725],{"type":45,"value":1726},"scripts\u002F",{"type":45,"value":1728}," subdirectory contains ready-to-run bash scripts for each analysis step. All scripts follow the output handling rules above (redirect to file, then process).",{"type":39,"tag":477,"props":1730,"children":1731},{},[1732,1746],{"type":39,"tag":481,"props":1733,"children":1734},{},[1735,1744],{"type":39,"tag":86,"props":1736,"children":1737},{},[1738],{"type":39,"tag":66,"props":1739,"children":1741},{"className":1740},[],[1742],{"type":45,"value":1743},"scripts\u002Fdiscover_schema.sh \u003CEXPERIMENT_ID> \u003CSESSION_ID>",{"type":45,"value":1745}," — Finds the first trace in the session, fetches its full detail, and prints the root span's attribute keys and input\u002Foutput values.",{"type":39,"tag":481,"props":1747,"children":1748},{},[1749,1758],{"type":39,"tag":86,"props":1750,"children":1751},{},[1752],{"type":39,"tag":66,"props":1753,"children":1755},{"className":1754},[],[1756],{"type":45,"value":1757},"scripts\u002Finspect_turn.sh \u003CTRACE_ID>",{"type":45,"value":1759}," — Fetches a specific trace, lists all spans, highlights error spans, and shows assessments.",{"type":39,"tag":48,"props":1761,"children":1763},{"id":1762},"example-wrong-answer-on-chat-turn-5",[1764],{"type":45,"value":1765},"Example: Wrong Answer on Chat Turn 5",{"type":39,"tag":55,"props":1767,"children":1768},{},[1769],{"type":45,"value":1770},"A user reports that their chatbot gave an incorrect answer on the 5th message of a chat conversation.",{"type":39,"tag":55,"props":1772,"children":1773},{},[1774],{"type":39,"tag":86,"props":1775,"children":1776},{},[1777],{"type":45,"value":1778},"1. Discover the schema and reconstruct the conversation.",{"type":39,"tag":55,"props":1780,"children":1781},{},[1782,1784,1789,1791,1796],{"type":45,"value":1783},"Fetch the first trace in the session and inspect the root span's attributes to find which keys hold inputs and outputs. In this case, ",{"type":39,"tag":66,"props":1785,"children":1787},{"className":1786},[],[1788],{"type":45,"value":496},{"type":45,"value":1790}," contains the user query and ",{"type":39,"tag":66,"props":1792,"children":1794},{"className":1793},[],[1795],{"type":45,"value":504},{"type":45,"value":1797}," contains the assistant response. Then search all session traces, extracting those fields in chronological order. Scanning the extracted inputs and outputs confirms that turn 5's response is wrong, and reveals whether earlier turns look correct.",{"type":39,"tag":55,"props":1799,"children":1800},{},[1801],{"type":39,"tag":86,"props":1802,"children":1803},{},[1804],{"type":45,"value":1805},"2. Check if the error originated in an earlier turn.",{"type":39,"tag":55,"props":1807,"children":1808},{},[1809],{"type":45,"value":1810},"Turn 3's response contains a factual error that the user didn't challenge. Turn 4 builds on that incorrect information, and turn 5 compounds it. The root cause is in turn 3, not turn 5.",{"type":39,"tag":55,"props":1812,"children":1813},{},[1814],{"type":39,"tag":86,"props":1815,"children":1816},{},[1817],{"type":45,"value":1818},"3. Analyze the root-cause turn as a single trace.",{"type":39,"tag":55,"props":1820,"children":1821},{},[1822],{"type":45,"value":1823},"Fetch the full trace for turn 3 and analyze it — examine assessments (if any), walk the span tree, check retriever results, and correlate with code. The retriever returned an outdated document, causing the wrong answer.",{"type":39,"tag":55,"props":1825,"children":1826},{},[1827],{"type":39,"tag":86,"props":1828,"children":1829},{},[1830],{"type":45,"value":1831},"4. Recommendations.",{"type":39,"tag":477,"props":1833,"children":1834},{},[1835,1840,1845],{"type":39,"tag":481,"props":1836,"children":1837},{},[1838],{"type":45,"value":1839},"Fix the retriever's data source to exclude or update outdated documents.",{"type":39,"tag":481,"props":1841,"children":1842},{},[1843],{"type":45,"value":1844},"Add per-turn assessments to detect errors before they propagate across the conversation.",{"type":39,"tag":481,"props":1846,"children":1847},{},[1848],{"type":45,"value":1849},"Consider implementing conversation-level error detection (e.g., checking consistency of answers across turns).",{"type":39,"tag":1851,"props":1852,"children":1853},"style",{},[1854],{"type":45,"value":1855},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1857,"total":2017},[1858,1872,1882,1899,1906,1917,1933,1951,1964,1976,1991,2002],{"slug":1859,"name":1859,"fn":1860,"description":1861,"org":1862,"tags":1863,"stars":1869,"repoUrl":1870,"updatedAt":1871},"setup","configure MLflow tracing","Configure MLflow tracing for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1864,1867,1868],{"name":1865,"slug":1866,"type":15},"Claude Code","claude-code",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},27014,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fmlflow","2026-07-14T05:39:00.297769",{"slug":1873,"name":1873,"fn":1874,"description":1875,"org":1876,"tags":1877,"stars":1869,"repoUrl":1870,"updatedAt":1881},"status","display MLflow tracing configuration","Show the current MLflow tracing configuration for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1878,1879,1880],{"name":1865,"slug":1866,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:39:01.540537",{"slug":1883,"name":1883,"fn":1884,"description":1885,"org":1886,"tags":1887,"stars":23,"repoUrl":24,"updatedAt":1898},"agent-evaluation","evaluate and optimize LLM agent output","Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong\u002Fincomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1888,1891,1894,1895],{"name":1889,"slug":1890,"type":15},"Agents","agents",{"name":1892,"slug":1893,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":1896,"slug":1897,"type":15},"Performance","performance","2026-07-14T05:39:15.600492",{"slug":4,"name":4,"fn":5,"description":6,"org":1900,"tags":1901,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1902,1903,1904,1905],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1907,"name":1907,"fn":1908,"description":1909,"org":1910,"tags":1911,"stars":23,"repoUrl":24,"updatedAt":1916},"analyzing-mlflow-trace","analyze MLflow traces","Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a trace ID and asks to debug, investigate, find issues, root-cause errors, understand behavior, or analyze quality. Triggers on \"analyze this trace\", \"what went wrong with this trace\", \"debug trace\", \"investigate trace\", \"why did this trace fail\", \"root cause this trace\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1912,1913,1914,1915],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:39:02.874441",{"slug":1918,"name":1918,"fn":1919,"description":1920,"org":1921,"tags":1922,"stars":23,"repoUrl":24,"updatedAt":1932},"fix-agent-issue","fix and update AI agent behavior","Drives a disciplined explore → plan → implement → verify loop for changing an AI agent's behavior with confidence — whether fixing a reported failure or introducing a new requirement, business rule, or policy. Grounds the diagnosis in MLflow traces, codifies the desired behavior as a regression test suite (`mlflow.genai.evaluate` assertions in `@mlflow.test` pytest tests), and iterates the agent — not the test — until green, resisting quick system-prompt patches when the real fix is upstream (missing tool, retrieval source, or capability). Use whenever the user wants to fix or change how an agent behaves — e.g. \"fix this issue in my agent\", \"this answer is wrong\", \"the agent is hallucinating\", \"improve my agent based on this trace\", \"make the agent do X instead of Y\", \"I want the agent to lead with\u002Fprioritize\u002Frecommend X\", \"new business rule: the agent should X\", \"always\u002Fnever do X\", \"change the agent's default behavior\" — or shares a trace they want addressed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1923,1924,1927,1928,1931],{"name":1889,"slug":1890,"type":15},{"name":1925,"slug":1926,"type":15},"Best Practices","best-practices",{"name":21,"slug":22,"type":15},{"name":1929,"slug":1930,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-30T05:53:39.749464",{"slug":1934,"name":1934,"fn":1935,"description":1936,"org":1937,"tags":1938,"stars":23,"repoUrl":24,"updatedAt":1950},"instrumenting-with-mlflow-tracing","instrument Python and TypeScript with MLflow Tracing","Instruments Python and TypeScript code with MLflow Tracing for observability. Must be loaded when setting up tracing as part of any workflow including agent evaluation. Triggers on adding tracing, instrumenting agents\u002FLLM apps, getting started with MLflow tracing, tracing specific frameworks (LangGraph, LangChain, OpenAI, Gemini, DSPy, CrewAI, AutoGen), or when another skill references tracing setup. Examples - \"How do I add tracing?\", \"Instrument my agent\", \"Trace my LangChain app\", \"Set up tracing for evaluation\"",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1939,1940,1941,1942,1943,1946,1947],{"name":1889,"slug":1890,"type":15},{"name":1892,"slug":1893,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":1944,"slug":1945,"type":15},"Python","python",{"name":13,"slug":14,"type":15},{"name":1948,"slug":1949,"type":15},"TypeScript","typescript","2026-07-20T05:58:52.968218",{"slug":1952,"name":1952,"fn":1953,"description":1954,"org":1955,"tags":1956,"stars":23,"repoUrl":24,"updatedAt":1963},"mlflow-agent","dispatch MLflow workflows and agent tasks","Master dispatcher for all MLflow workflows. Use this skill when the user wants to do anything with MLflow — tracing, evaluating, debugging, or improving an agent. Routes to the right MLflow sub-skill automatically. Triggers on: \"use mlflow\", \"help with mlflow\", \"mlflow agent\", \"add mlflow to my project\", \"trace my agent\", \"evaluate my agent\", or any MLflow task without a specific skill in mind.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1957,1958,1959,1962],{"name":1889,"slug":1890,"type":15},{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},"MLOps","mlops",{"name":13,"slug":14,"type":15},"2026-07-14T05:39:04.133424",{"slug":1965,"name":1965,"fn":1966,"description":1967,"org":1968,"tags":1969,"stars":23,"repoUrl":24,"updatedAt":1975},"mlflow-onboarding","onboard users to MLflow workflows","Onboards users to MLflow by determining their use case (GenAI agents\u002Fapps or traditional ML\u002Fdeep learning) and guiding them through relevant quickstart tutorials and initial integration. If an experiment ID is available, it should be supplied as input to help determine the use case. Use when the user asks to get started with MLflow, set up tracking, add observability, or integrate MLflow into their project. Triggers on \"get started with MLflow\", \"set up MLflow\", \"onboard to MLflow\", \"add MLflow to my project\", \"how do I use MLflow\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1970,1971,1972],{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},{"name":1973,"slug":1974,"type":15},"Onboarding","onboarding","2026-07-14T05:39:06.681003",{"slug":1977,"name":1977,"fn":1978,"description":1979,"org":1980,"tags":1981,"stars":23,"repoUrl":24,"updatedAt":1990},"querying-mlflow-metrics","fetch trace metrics from MLflow servers","Fetches aggregated trace metrics (token usage, latency, trace counts, quality evaluations) from MLflow tracking servers. Triggers on requests to show metrics, analyze token usage, view LLM costs, check usage trends, or query trace statistics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1982,1985,1988,1989],{"name":1983,"slug":1984,"type":15},"Analytics","analytics",{"name":1986,"slug":1987,"type":15},"Metrics","metrics",{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},"2026-07-14T05:39:13.07056",{"slug":1992,"name":1992,"fn":1993,"description":1994,"org":1995,"tags":1996,"stars":23,"repoUrl":24,"updatedAt":2001},"retrieving-mlflow-traces","retrieve and query MLflow traces","Retrieves MLflow traces using CLI or Python API. Use when the user asks to get a trace by ID, find traces, filter traces by status\u002Ftags\u002Fmetadata\u002Fexecution time, query traces, or debug failed traces. Triggers on \"get trace\", \"search traces\", \"find failed traces\", \"filter traces by\", \"traces slower than\", \"query MLflow traces\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1997,1998,1999,2000],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:39:09.22888",{"slug":2003,"name":2003,"fn":2004,"description":2005,"org":2006,"tags":2007,"stars":23,"repoUrl":24,"updatedAt":2016},"sagemaker-mlflow","connect to SageMaker Managed MLflow","Connect to SageMaker Managed MLflow (mlflow-app or mlflow-tracking-server ARN) as an MLflow backend, then hand off to the other MLflow skills. Triggers on a SageMaker MLflow ARN (arn:aws:sagemaker:...:mlflow-app\u002F... or arn:aws:sagemaker:...:mlflow-tracking-server\u002F...) or \"SageMaker Managed MLflow\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2008,2011,2014,2015],{"name":2009,"slug":2010,"type":15},"AI Infrastructure","ai-infrastructure",{"name":2012,"slug":2013,"type":15},"AWS","aws",{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},"2026-07-14T05:39:05.401801",13,{"items":2019,"total":340},[2020,2027,2034,2041,2049,2059,2066],{"slug":1883,"name":1883,"fn":1884,"description":1885,"org":2021,"tags":2022,"stars":23,"repoUrl":24,"updatedAt":1898},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2023,2024,2025,2026],{"name":1889,"slug":1890,"type":15},{"name":1892,"slug":1893,"type":15},{"name":9,"slug":8,"type":15},{"name":1896,"slug":1897,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2028,"tags":2029,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2030,2031,2032,2033],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1907,"name":1907,"fn":1908,"description":1909,"org":2035,"tags":2036,"stars":23,"repoUrl":24,"updatedAt":1916},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2037,2038,2039,2040],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1918,"name":1918,"fn":1919,"description":1920,"org":2042,"tags":2043,"stars":23,"repoUrl":24,"updatedAt":1932},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2044,2045,2046,2047,2048],{"name":1889,"slug":1890,"type":15},{"name":1925,"slug":1926,"type":15},{"name":21,"slug":22,"type":15},{"name":1929,"slug":1930,"type":15},{"name":9,"slug":8,"type":15},{"slug":1934,"name":1934,"fn":1935,"description":1936,"org":2050,"tags":2051,"stars":23,"repoUrl":24,"updatedAt":1950},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2052,2053,2054,2055,2056,2057,2058],{"name":1889,"slug":1890,"type":15},{"name":1892,"slug":1893,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"name":1944,"slug":1945,"type":15},{"name":13,"slug":14,"type":15},{"name":1948,"slug":1949,"type":15},{"slug":1952,"name":1952,"fn":1953,"description":1954,"org":2060,"tags":2061,"stars":23,"repoUrl":24,"updatedAt":1963},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2062,2063,2064,2065],{"name":1889,"slug":1890,"type":15},{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},{"name":13,"slug":14,"type":15},{"slug":1965,"name":1965,"fn":1966,"description":1967,"org":2067,"tags":2068,"stars":23,"repoUrl":24,"updatedAt":1975},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2069,2070,2071],{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},{"name":1973,"slug":1974,"type":15}]