[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-qwen-cua-driver-rs":3,"mdc--qpt0c2-key":37,"related-org-qwen-cua-driver-rs":5253,"related-repo-qwen-cua-driver-rs":5424},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"cua-driver-rs","drive native GUI applications via MCP","Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server — snapshot its accessibility tree, click\u002Ftype\u002Fscroll by element_index or pixel coords, verify via re-snapshot, all without bringing the target to the foreground. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"qwen","Qwen","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fqwen.png","QwenLM",[13,17,20,23],{"name":14,"slug":15,"type":16},"Accessibility","accessibility","tag",{"name":18,"slug":19,"type":16},"MCP","mcp",{"name":21,"slug":22,"type":16},"Desktop","desktop",{"name":24,"slug":25,"type":16},"Browser Automation","browser-automation",26008,"https:\u002F\u002Fgithub.com\u002FQwenLM\u002Fqwen-code","2026-07-16T05:59:28.687299",null,2643,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"An open-source AI coding agent that lives in your terminal.","https:\u002F\u002Fgithub.com\u002FQwenLM\u002Fqwen-code\u002Ftree\u002FHEAD\u002Fpackages\u002Fcua-driver\u002Frust\u002FSkills\u002Fcua-driver","---\nname: cua-driver-rs\ndescription: Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server — snapshot its accessibility tree, click\u002Ftype\u002Fscroll by element_index or pixel coords, verify via re-snapshot, all without bringing the target to the foreground. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host.\n---\n\n# cua-driver-rs\n\nOrchestrates cross-platform app automation via `cua-driver`. Whenever\na user asks to drive a native app, follow the loop in this skill\nrather than calling tools ad-hoc — the snapshot-before-action\ninvariant is not optional and silently breaks if you skip it.\n\n## Platform-specific reading — read this first\n\nThis file is the **cross-platform core**: snapshot invariant, CLI vs\nMCP choice, tool surface naming, behavior matrix, canonical loop,\npixel-click contract, common failure modes. The platform-specific\nmaterial (forbidden-list, accessibility tree implementation, launch\nsemantics, click dispatch) lives in companion files in this same\ndirectory:\n\n- **macOS** — read `MACOS.md` (no-foreground contract, forbidden\n  `open`\u002F`osascript`\u002F`cliclick` invocations, AXMenuBar navigation,\n  SkyLight pixel-click dispatch, Apple-Events JS bridge).\n- **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP \u002F\n  ApplicationFrameHost hosting, layered UIA+PostMessage click chain,\n  Session 0 isolation, Windows-specific focus-steal vectors).\n- **Linux** — read `LINUX.md` (X11 background input via AT-SPI +\n  XSendEvent, recording, Wayland opt-in\u002Fpreview).\n\nCross-cutting topics also have their own files:\n\n- `WEB_APPS.md` — browser \u002F Electron \u002F Tauri specifics (sparse AX\n  trees, omnibox navigation, the `set_value` workaround for\n  minimized windows, tabs-vs-windows guidance).\n- `RECORDING.md` — session recording + `replay_trajectory`.\n- `TESTS.md` — internal test surface.\n\nUse whichever combination matches the host. When in doubt, run\n`cua-driver doctor` — it reports the platform and the right entry\npoint.\n\n## The no-foreground principle (cross-platform)\n\n**The user's frontmost app MUST NOT change.** Every platform has its\nown list of forbidden commands; the principle is universal:\n\n- macOS: any `open` invocation, any `osascript` that mutates GUI\n  state, `cliclick`, `cghidEventTap` writes targeting another app's\n  window. Full list in `MACOS.md`.\n- Windows: any `Start-Process` that triggers a `ShowWindow`\u002F`SetForegroundWindow`\n  on the target, `WScript.Shell.AppActivate`, attaching to the\n  foreground thread for input forwarding. Full list in `WINDOWS.md`.\n\nIf you reach for a command that says \"activate\", \"foreground\",\n\"raise\", or \"make key\", stop and translate to the cua-driver tool\nthat does the same intent without focus-stealing.\n\n## Defaults — always prefer cua-driver over shell shims\n\n**Default transport is the `cua-driver` CLI** — `Bash` shelling out\nto `cua-driver \u003Ctool-name> '\u003CJSON-args>'`. MCP tools (prefix\n`mcp__cua-driver__*`) only when the user explicitly asks for them.\nCLI wins because it picks up rebuilds instantly, failures are\neasier to diagnose, and there's no per-tool schema-load overhead.\n\nEvery reference to `click(...)`, `get_window_state(...)` etc. in this\nskill means `cua-driver click '{...}'` — translate to MCP form only\nwhen MCP is requested.\n\n### Claude Code computer-use compatibility mode\n\nFor normal Claude Code use, keep the default CLI or `cua-driver` MCP\nserver path above. If the user explicitly wants Claude Code's\nvision\u002Fcomputer-use-style flow, they can register:\n\n```bash\ncua-driver mcp-config --client claude   # then paste + run the printed line\n```\n\nObservation: Claude Code vision flows appear to treat a screenshot\nMCP tool as the image-grounding anchor. This compatibility mode keeps\nthe normal CuaDriver tools and changes only `screenshot`. The\ncompatibility `screenshot` requires `pid` and `window_id`, captures\nonly that target window, and returns the window-local pixel\ncoordinate frame. Start with `launch_app` or `list_windows`, then\ncall `screenshot({pid, window_id})`; do not assume desktop\ncoordinates or a full-screen capture.\n\nUse MCP for this Claude Code vision\u002Fcomputer-use-style path. Do not\nshell out to `cua-driver screenshot` as a substitute: CLI screenshots\nstill work as CuaDriver calls, but they do not expose the\n`mcp__cua-computer-use__screenshot` tool name that Claude Code\nappears to use as the image-grounding cue.\n\n## Using cua-driver from the shell\n\nTool names are `snake_case`, management subcommands are\n`kebab-case` — no ambiguity. Tools invoked as `cua-driver\n\u003Ctool-name> '\u003CJSON-args>'`. Management subcommands:\n\n- `cua-driver serve` — start persistent daemon (**required** for\n  `element_index` workflows; without it each CLI invocation spawns a\n  fresh process and the per-pid element cache dies between calls).\n  macOS users: see `MACOS.md` for the LaunchServices-routed launch\n  form.\n- `cua-driver stop` \u002F `status`\n- `cua-driver list-tools`, `describe \u003Ctool>`\n- `cua-driver recording start|stop|status` — see `RECORDING.md`\n- `cua-driver check-update [--json] [--no-cache]` — read-only \"is a newer release available?\" probe. Same payload as the `check_for_update` MCP tool; pair with `cua-driver update --apply` to install.\n\nCanonical multi-step workflow (example shape — platform-specific\nlaunch idioms in the per-OS companion file):\n\n```bash\ncua-driver serve\ncua-driver launch_app '{\"bundle_id\":\"...\"}'\n# → {pid: 844, windows: [{window_id: 10725, ...}]}\ncua-driver get_window_state '{\"pid\":844,\"window_id\":10725}'\ncua-driver click '{\"pid\":844,\"window_id\":10725,\"element_index\":14}'\ncua-driver stop\n```\n\n## Agent cursor overlay\n\nVisual cursor overlay for demos and screen recordings. Default:\nenabled — you do NOT need to enable it. Toggle with\n`cua-driver set_agent_cursor_enabled '{\"enabled\":true|false}'` only to\nhide or re-show it. A triangle pointer Bezier-glides to each click\ntarget, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs:\n`set_agent_cursor_motion` takes any subset of `start_handle`,\n`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,\npersisted to config.\n\n**Per-session cursors.** Each MCP session automatically owns its own\ncursor, keyed by the session's id (the proxy mints one session id per\nMCP connection and the daemon scopes the cursor, config overrides, and\nrecording to it). You normally pass nothing — the session key is wired\nthrough for you. Pass an explicit `cursor_id` only to *deliberately\nshare* one cursor across sessions. When a session ends (the MCP client\ndisconnects) its cursor is removed automatically.\n\n**Visibility caveat (AX runs).** On a pure accessibility-action run\n(clicking by `element_index`), the first action **seeds the cursor\non-screen a short distance from the target and plays a brief glide +\npulse** — not the long Bezier sweep a cursor already on-screen would\ntrace from its previous spot. It's subtle and easy to miss in a\nrecording. If you want a clearly *gliding* cursor for a demo or screen\nrecording, do a pixel click (`click({pid,x,y})`) or a `move_agent_cursor`\nfirst to put the cursor on-screen; subsequent AX actions then glide the\nfull path normally.\n\nRequires the daemon process's UI runloop, which `cua-driver serve` \u002F\n`mcp` bootstraps. One-shot CLI invocations skip the overlay entirely.\n\n## The core invariant — snapshot before AND after every action\n\n**Every action MUST be bracketed by `get_window_state(pid, window_id)`**:\n\n- **Before** — the pre-action snapshot resolves the `element_index`\n  you're about to use. Indices from previous turns are stale; the\n  server replaces the element index map on every snapshot, keyed\n  on `(pid, window_id)`. Indices from turn N don't resolve in turn\n  N+1, and indices from window A don't resolve against window B of\n  the same app. Skip this and element-indexed actions fail with\n  `No cached AX state`.\n- **After** — the post-action snapshot verifies the action actually\n  landed. Without it you can't tell a silent no-op from a real\n  effect. The accessibility-tree change (new value, new window,\n  disappeared menu, disabled button, etc.) is your evidence that\n  the action fired. If nothing changed, the action probably failed\n  silently — say so, don't assume success.\n\nThis applies to pixel clicks too — re-snapshot after to confirm the\nclick landed on the intended target.\n\n### Why window selection is the caller's job now\n\n`get_app_state` used to pick a window for you via a max-area heuristic\nthat returned the wrong surface on apps with large off-screen utility\npanels. Concrete reproducer: IINA's OpenSubtitles helper (600×432\noff-screen) out-area'd the visible 320×240 player window, so\n`get_app_state(pid)` screenshot'd the invisible panel and clicks landed\nthere silently. The new `get_window_state(pid, window_id)` makes the\ncaller name the window explicitly — the driver validates that the\nwindow belongs to the pid and is on the current Space\u002Fdesktop, then\nsnapshots exactly what was asked for. Enumerate candidates via\n`list_windows` or read the `windows` array `launch_app` already\nreturns.\n\n## Behavior matrix\n\n### Perception is mode-agnostic — `get_window_state` returns BOTH\n\n`get_window_state(pid, window_id)` **returns both the accessibility\ntree AND a screenshot by default.** There is no capture mode to pick\nand nothing to configure — you ground on the tree and the screenshot\ntogether, and you cross-check one against the other. This matters\nbecause the tree **lies** on some surfaces:\n\n- **Electron** echo-confirms a `set_value` \u002F `type_text` against the AX\n  shim while the rendered text view never changed.\n- **Catalyst** (iOSAppOnMac) exposes null \u002F placeholder `AXValue`s.\n- **Virtualized \u002F off-viewport list rows** report bogus frames (an\n  `h:1` height, an off-screen origin) for rows that aren't actually\n  laid out.\n\nA grounding screenshot is present by default, so when the tree looks\nwrong you look at the pixels **in the same response** — no second\ncapture, no mode flip.\n\n> **Perf opt-out — `include_screenshot`.** `include_screenshot`\n> (boolean, default `true`) is the one knob, and it is a **perf** knob,\n> not a modality choice. Default returns both (grounding-first). Pass\n> `include_screenshot:false` to skip the screen grab and get the tree\n> only — the cheap path when you're just **re-indexing before an\n> element ax action** and don't need to re-ground on pixels. The\n> `ax`\u002F`px` decision still lives at action time, not here.\n\n> **`capture_mode` is DEPRECATED and ignored.** It is still *accepted*\n> on `get_window_state` so old callers don't error, but it has **no\n> effect** — both the tree and the screenshot come back regardless of\n> what you pass (`ax`, `vision`, `som`, anything). There is no\n> `ax`\u002F`vision`\u002F`som` capture choice anymore. Drop the word \"vision\"\n> for perception entirely. (The tool named `screenshot` is separate —\n> raw PNG, no AX walk — and unrelated.)\n\n### The modality is chosen at ACTION time — `ax` vs `px`\n\nYou don't pick a capture mode; you pick **how you address the target**\non the action call, and that one choice selects the rung:\n\n- **element ax action** — pass `element_index` \u002F `element_token`.\n  Dispatches through the **accessibility rung**: AXPress (macOS) \u002F UIA\n  Invoke (Windows) \u002F AT-SPI `doAction` (Linux). Backgroundable,\n  z-order-independent, and the only **driver-verifiable** rung.\n- **element px action** — pass `x`, `y`. Dispatches through the **pixel\n  rung**, reading the coordinate straight off the screenshot that's\n  already in the `get_window_state` response. Best-effort; the caller\n  confirms the effect.\n\n`ax`↔`element_index`, `px`↔pixel `x,y`. We retired the word \"vision\"\nfor the *dispatch* path — it conflated perception with dispatch.\nPerception is always both; dispatch is `ax` or `px`.\n\n**The keyboard family has both forms too.** `type_text`, `press_key`,\nand `hotkey` take `element_index` (ax) **or** `x,y` (px) — mutually\nexclusive, same as the pointer tools. The px form **pixel-clicks at\n`(x,y)` to establish real renderer focus, then delivers the\nkeystroke(s)** to the now-focused element (it reuses `click`'s\ncoordinate translation + `delivery_mode`). That gives e.g.\n`type_text({pid, window_id, x, y, text})` as a one-call focus-then-type\nfor Chromium\u002FElectron inputs the AX path can't reach, and\n`hotkey({pid, x, y, keys:[\"cmd\",\"v\"]})` to paste into a specific field.\n\n**Typing default (the ladder).** Call `type_text` directly with\n`element_index` (ax) — it targets the field, no pre-click. On\nElectron\u002FCatalyst the AX layer echoes the write without rendering it,\nso the driver returns `effect:\"unverifiable\"` + `escalation:\"px\"`\nthere (never a false `verified:true`) — follow it, and cross-check the\nscreenshot in the response (the only ground truth). Escalate to the px\nform — `type_text({pid, window_id, x, y, text})` — which pixel-clicks\nto focus, then types. **If the target control is closed** (a search\nbutton, a collapsed field), AX-press to open it first (AX actions work\nin the background): a px focus-click won't reliably open *and* focus a\nclosed control, so the text leaks into whatever's already focused.\nEscalate to `delivery_mode:\"foreground\"` only if it still drops.\n\n**`set_value` stays AX-only by design** — it's for **non-text**\ncontrols (dropdown \u002F `AXPopUpButton`, checkbox, slider, stepper). Its\npixel counterpart is a `click`\u002F`drag` on the control, not a \"set value\nat a pixel.\" So: text → `type_text` (ax+px); non-text control values →\n`set_value`; pixel-manipulate a control → `click`\u002F`drag`.\n\n**Action responses carry an effect\u002Fescalation verdict**\n\nEvery action response keeps `verified` (did the driver read back a\npost-condition?) and adds two machine-readable fields so you know\nwhether — and where — to climb the ladder:\n\n- `effect`: one of\n  - `\"confirmed\"` — the driver read back the effect (`ax` rung only).\n  - `\"unverifiable\"` — dispatched, but the driver has no handle to\n    read back (every `px`\u002FCGEvent path; foreground rung). **You**\n    confirm it off the screenshot — it is not a failure.\n  - `\"suspected_noop\"` — the `ax` action **likely did nothing** (the\n    element didn't actually advertise the action, or you hit a passive\n    label). This is the explicit **\"cross to `px`\"** trigger.\n- `escalation`: `{recommended, reason}` when the driver thinks you\n  should change rung —\n  - `\"px\"` — the element isn't really actionable in `ax`; do an\n    **element px action** off the screenshot you already have.\n  - `\"foreground\"` — a background insert\u002Fclick was *dropped* on\n    delivery; re-call the same action with `delivery_mode:\"foreground\"`.\n\n`get_window_state` itself, when the AX tree comes back empty (a non-AX\nsurface like Electron\u002FChromium\u002Fcanvas), returns `degraded: true`\n**plus the same `escalation` hint** — normally pointing at `px` (you\nstill have the screenshot from the same call to click off).\n\n**Platform nuance for `escalation`.** On **Wayland** an unfocused\nwindow cannot be pixel-targeted in the background (libei →\n`background_unavailable`), so there the recommendation is\n**`foreground`, not `px`**. macOS, X11, and most Windows surfaces\n*can* pixel-target in the background, so they recommend `px`. See\n`LINUX.md` \u002F `WINDOWS.md`.\n\n## The verify-then-escalate ladder (algorithm)\n\nEvery snapshot already hands you both the tree and the screenshot, so\nverifying never means \"go take a screenshot\" — it means cross-check\nthe tree against the pixels you already have, and only change\n*dispatch rung* on a real signal. Walk the rungs:\n\n```\n# Rung 1 — element ax action, backgrounded (the cheap default)\nget_window_state(pid, window_id)            # tree + screenshot, both, always\nresp = click(pid, window_id, element_index) # or type_text \u002F set_value \u002F press_key\nget_window_state(pid, window_id)            # re-snapshot — did the tree change?\n\nif resp.effect == \"confirmed\" and tree changed:\n    done                                    # driver-verified\n\n# escalate only on a real signal\nif resp.effect == \"suspected_noop\"\n   or resp.escalation.recommended == \"px\"\n   or get_window_state.degraded            # empty tree → non-AX surface\n   or the tree looks wrong vs the screenshot:   # e.g. an h:1 \u002F off-viewport row\n\n    # Rung 2 — element px action off the SAME screenshot\n    pick the target pixel from the screenshot already in the response\n    click(pid, x, y)                        # background pixel — still no foreground\n    get_window_state(pid, window_id)        # re-snapshot, eyeball the result\n    if it landed: done\n\n# Rung 3 — background delivery was dropped (insert\u002Fclick never arrived)\nif resp.escalation.recommended == \"foreground\"\n   or the px action still did nothing:\n    re-call the same action with delivery_mode:\"foreground\"\n    # on Wayland this is the ONLY escalation — px-bg can't target an\n    # unfocused window there; see LINUX.md\n    verify again\n```\n\nThe two ideas to hold onto: (1) the AX tree **lies** on canvas \u002F web \u002F\nCatalyst \u002F virtualized surfaces, so an unchanged-or-bogus tree plus\n`suspected_noop`\u002F`degraded` — or a tree that simply disagrees with the\nscreenshot — is your cue to do an **element px action** off the\nscreenshot you already have; (2) `px` is a *conscious* switch to the\npixel addressing path, not a different capture.\n\n**Window state → what works**\n\n| state | `get_window_state` | element-index click (AX\u002FUIA) | `press_key` commit | pixel click |\n|---|---|---|---|---|\n| frontmost | ✅ | ✅ | ✅ | ✅ |\n| backgrounded \u002F visible | ✅ | ✅ | ✅ | ✅ |\n| **minimized** | ✅ | ✅ (actions fire in place) | ❌ silent no-op — use `set_value` or click equivalent | ❌ no on-screen bounds |\n| hidden | ✅ | ✅ | depends | ❌ |\n| on another desktop \u002F Space | ⚠️ tree may be stripped on some apps — response carries `off_space: true` so you can detect it | ✅ | ✅ | ❌ not in current-desktop list |\n\n**Critical cell — minimized + keyboard commit.** The keystroke\nreaches the app but accessibility focus doesn't propagate to renderer\nfocus on a minimized window. Workarounds in order of preference:\n`set_value` to write the field's entire value directly, or\nelement-index-click a commit-equivalent button (Go, Submit,\ncheckbox). Tell the user the window needs to un-minimize only as a\nlast resort.\n\n## The canonical loop\n\n```\nstart_session(session)            # once per run: declares this run's identity\nlaunch_app(target)\n  → pick window_id from the returned `windows` array\n    (or call list_windows(pid) separately)\n  → get_window_state(pid, window_id)\n    → [act]  # every action also takes (pid, window_id) + your `session`\n  → get_window_state(pid, window_id) → verify\nend_session(session)              # when the run finishes\n```\n\n`launch_app` now returns a `windows` array alongside the pid, so the\ncommon case collapses to two calls (`launch_app` → `get_window_state`)\nwithout a separate `list_windows` hop.\n\n**Declare a session.** A session is *your run's* identity — a stable id\nyou choose (`\"research-1\"`), declared with `start_session` and passed as\n`session` on every action. It owns your agent cursor (a distinct colour\nper id), follows the run across any apps\u002Fwindows, and is the same whether\nyou drive over MCP, the CLI, or the socket. The cursor is **opt-in**: it\nappears only once you declare a session (anonymous actions run cursor-less).\nEnd with `end_session` (or the idle-TTL reclaims it).\n\n**Concurrent runs\u002Fsubagents:** `launch_app` is idempotent — two runs that\nlaunch the same app get the **same** instance (and on single-instance apps\nlike Calculator, the same window), so they clobber each other. Give each run\nits **own `session`** (→ its own cursor) AND pass\n`creates_new_application_instance: true` to `launch_app` (→ its own window).\nThe element cache is keyed on `(pid, window_id)` and the cursor on `session`,\nso distinct instances + distinct sessions keep the runs fully separated.\n\n**Parallelism vs. ordering.** Distinct sessions give distinct *cursors*, not\ndistinct *connections*. Subagents that share one `cua-driver mcp` (stdio)\nconnection have their tool calls **serialized** by the transport — they take\nturns, not run in parallel. That's not a correctness problem (session + window\nisolation means they can't collide), just a throughput one. For genuinely\nparallel agents, give each its **own connection**: separate `cua-driver mcp`\nprocesses, or point each agent's MCP client at the daemon's HTTP endpoint\n(`CUA_DRIVER_RS_MCP_HTTP_PORT` → `POST http:\u002F\u002F127.0.0.1:\u003Cport>\u002Fmcp`). The daemon\nserves connections concurrently; per-connection ordering keeps each agent's own\nsequence (e.g. `3 → + → 1 → =`) correct.\n\n`list_apps` is for app-level discovery (answering \"what's installed \u002F\nrunning \u002F frontmost?\") — not part of the core action loop. Skip it\nin the loop. For **window-level** questions — \"does this app have a\nvisible window?\", \"which desktop is this window on?\", \"which of this\npid's windows is the main one?\" — call `list_windows` instead; the\napp record doesn't carry window state on purpose. In the common\nsingle-window case you can skip `list_windows` entirely and read the\n`windows` array that `launch_app` already returned.\n\n### Snapshot and act by element_index\n\nCall `get_window_state({pid, window_id})` with the `window_id` from\n`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if\nyou're interacting with a long-lived process). It returns **the tree\nand the screenshot together** by default, so you can both dispatch by\n`element_index` and ground on pixels from one call — no config change,\nno mode flip. When you're just re-indexing before an element ax action\nand don't need fresh pixels, pass `include_screenshot:false` to skip\nthe grab (a perf knob, not a modality choice).\n\nThe response carries:\n\n- `tree_markdown` — every actionable element tagged `[N]`. That `N`\n  is the `element_index`. The tree can be very large (Finder is\n  ~1600 elements, ~190 KB); when it exceeds token limits the MCP\n  harness saves it to a file and returns the path. Use `Bash` +\n  `jq -r '.tree_markdown'` + `grep` to pull the section you need.\n- `effect` \u002F `escalation` \u002F `degraded` — the verify-then-escalate\n  signals (see the behavior matrix above): `degraded: true` means the\n  tree came back empty (non-AX surface), so you act by **`px`** off the\n  screenshot in the same response.\n- `screenshot_file_path` — present when the screenshot was written to\n  disk instead of inlined (you passed `screenshot_out_file`, or the\n  context-saving CLI path); otherwise the frame is inlined.\n- `screenshot_width` \u002F `_height` \u002F `_scale_factor` — dimensions of\n  the captured image. Present whenever a screenshot was taken (i.e.\n  unless you passed `include_screenshot:false`).\n\n**Getting the screenshot as a file (CLI and context-constrained agents):**\n\n```bash\n# write to file — stdout stays readable (AX\u002FUIA tree \u002F summary only, no base64)\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W,\"screenshot_out_file\":\"\u002Ftmp\u002Fshot.jpg\"}'\n\n# CLI --screenshot-out-file flag is equivalent\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W}' --screenshot-out-file \u002Ftmp\u002Fshot.jpg\n```\n\nPass `screenshot_out_file` when using `get_window_state` via CLI or\nfrom an agent whose context window can't absorb ~31 KB of inline\nbase64 (e.g. OpenCode with a local Ollama model). The MCP image\ncontent block is omitted from the response when this param is set —\nthe model receives only the tree and `screenshot_file_path`, then\nreads the image from disk.\n\n**The tree and the screenshot are complementary, not redundant — and\nthey come from the *same* call.** Each half carries signal the other\ncan't, which is exactly why you cross-check them:\n\n- The **tree** tells you *what's clickable* — roles, labels,\n  `element_index` handles, advertised actions, parent-child\n  structure. This is the ground truth for an **element ax action**.\n- The **screenshot** tells you *which one* — the tree often has many\n  buttons with similar or empty labels (\"Delete\", \"OK\", anonymous\n  UUID-labeled buttons, repeated static-text), and visual context\n  disambiguates. Captions, colors, layout relationships visible in\n  pixels often don't show up in the tree at all (especially in\n  Chromium \u002F Electron \u002F web content) — and the screenshot is where you\n  catch the tree *lying* (an `h:1`\u002Foff-viewport row, a Catalyst null\n  value).\n\nDefault to dispatching by `element_index` (the **element ax action**) —\nit's the verifiable, backgroundable rung. Do an **element px action**\n(`x,y` off the same screenshot) when the tree can't disambiguate\n(repeated\u002Fempty labels), when it's empty (`degraded` — non-AX\nsurface), when an action came back `suspected_noop`, or when the tree\ndisagrees with the pixels. You never re-capture to switch — the\nscreenshot is already there; you just change *how you address* the\ntarget.\n\nReach for pixel coordinates only when the target is a canvas \u002F\nvideo \u002F WebGL \u002F custom-drawn surface that isn't in the tree\n(see \"Pixel-coordinate clicks\" below).\n\nThe `actions=[...]` list on each element is **advisory**, not\nauthoritative. cua-driver does not pre-flight check against it —\n`click({pid, element_index})` always attempts the default action (or\nthe action you pass) and surfaces whatever the target returns. **Try\nthe click first** — pivot only on the returned error code.\n\n### Tool dispatch table\n\nEvery row assumes a `(pid, window_id)` pair from the last\n`get_window_state`; `window_id` is required alongside `element_index`,\nignored on pixel-only forms unless you want to anchor the conversion\nagainst a specific window.\n\n| Intent | Tool | Notes |\n|---|---|---|\n| List an app's windows | `list_windows({pid})` | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids |\n| Snapshot a window | `get_window_state({pid, window_id})` | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache |\n| Left click | `click({pid, window_id, element_index})` | default `action: \"press\"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: [\"cmd\"\\|\"ctrl\"]` |\n| Double-click \u002F open | `double_click({pid, window_id, element_index})` | Default action when the element advertises one (Open on Finder items \u002F openable rows), else stamped pixel double-click at the element's center |\n| Right click \u002F context menu | `right_click({pid, window_id, element_index})` or `click({pid, window_id, element_index, action: \"show_menu\"})` | Chromium web-content coerces pixel right-click to left on macOS — see `WEB_APPS.md` |\n| Type at cursor | `type_text({pid, text, window_id, element_index})` (ax) or `type_text({pid, text, window_id, x, y})` (px) | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium\u002FElectron inputs the AX path can't reach |\n| Set whole non-text control value | `set_value({pid, window_id, element_index, value})` | **AX-only by design** — dropdown\u002F`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`\u002F`drag` |\n| Scroll | `scroll({pid, direction, amount, by, window_id, element_index})` | synthesizes per-pid PageUp\u002FPageDown\u002Farrows |\n| Focus + send key | `press_key({pid, key, window_id, element_index, modifiers})` (ax) or `press_key({pid, key, x, y})` (px) | ax `element_index` sets focus then posts the key; **px** pixel-clicks `(x,y)` to focus, then sends the key |\n| Send key to pid | `press_key({pid, key, modifiers})` | no focus change; key goes to pid's current focus |\n| Modifier combo | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px) | e.g. `[\"cmd\",\"c\"]` \u002F `[\"ctrl\",\"c\"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `[\"cmd\",\"v\"]` to paste into it |\n\n**All keyboard\u002Ftext primitives require `pid`.** There is no\nfrontmost-routed variant — every key goes to the named target via\nthe platform's per-pid event-post path, so the driver cannot leak\nkeystrokes into the user's foreground app.\n\n**Why `element_index` is the primary path:** works on hidden \u002F\noccluded \u002F off-desktop windows, no focus steal, stable across\nrebuilds, labels tell you what you're clicking. Reach for pixel\ncoordinates only when the accessibility tree can't.\n\n## Cross-platform parameter contract\n\nThe capture, dispatch, and addressing params — `session`,\n`delivery_mode`, `capture_mode` (deprecated\u002Fignored — see the behavior\nmatrix; still in the schema only so old callers don't error), `scope`,\n`modifier`, `button`, `element_index`, `element_token` — are a **shared\nschema contract**: identical *shape* (`type`\u002F`enum`\u002F`items`) on macOS,\nWindows, and Linux.\nThey compose from canonical fragments in\n`cua-driver-core::tool_schema` (+ `capture_mode`), and a CI gate\n(`schema_consistency_test`) runs every tool's live `tools\u002Flist` through a\nstructural checker on each platform, so the three surfaces can't\nsilently drift. *Contributor note:* when you add or edit one of these\nshared params on a tool, pull from the fragment — don't re-hand-write the\nJSON, or the gate fails. (Descriptions may legitimately vary per tool;\nthe gate compares shape, not prose.)\n\nTwo consequences for callers:\n\n- **`session` is accepted on every action and cursor tool, on all three\n  platforms.** It's cursor-wired where the platform glides a cursor and\n  schema-accepted everywhere else — so the same `session` you pass on\n  macOS is no longer *rejected* by Windows\u002FLinux, which previously\n  refused unknown keys via `additionalProperties:false`.\n- **`delivery_mode` (`\"background\"` default \u002F `\"foreground\"`) is on the\n  whole input family** — `click`, `double_click`, `right_click`, `drag`,\n  `scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The\n  `foreground` rung briefly fronts the target, acts, then restores the\n  prior frontmost: the explicit last resort when a background attempt\n  didn't land. **`foreground` is a reaction, never a prediction.** Always\n  fire the `background` default first and let the driver tell you it\n  can't (a `background_unavailable` error or `escalation.recommended ==\n  \"foreground\"`) — or observe a verified no-op — *before* you escalate.\n  Do **not** reason \"it's a GTK\u002FChromium\u002FElectron app, so background will\n  drop, so I'll front up-front\": the toolkit lists in the tool schemas\n  are the *driver's* internal detectors, not a checklist for you to front\n  on a guess. (Concretely: GIMP's GTK toolbox accepts background pixel\n  clicks fine — a preemptive foreground click there just steals the\n  user's focus for nothing.) What each platform's *background* rung can\n  actually carry differs (e.g. a Windows background click can't carry\n  `modifier` state — see `WINDOWS.md`); the schema is uniform, the\n  residual limits are per-OS.\n\n**Required-set contract.** `click` requires nothing (`required:[]`),\n`scroll` requires `[\"direction\"]`, `zoom` requires\n`[\"window_id\",\"x1\",\"y1\",\"x2\",\"y2\"]` — same on every platform. `pid` is\n**conditionally** required (needed unless a windowless desktop-scope\ncall) and validated in code with a clear error, NOT pinned in the schema\n— so omitting `pid` for a desktop-scope action is no longer\nschema-rejected.\n\nGenuinely platform-specific params stay OUT of the shared contract by\ndesign (launch-app identifiers, the Windows-only `debug_window_info`, the\nmacOS-only `check_permissions.prompt`). The per-OS files list the\nresiduals that matter when you drive on that platform.\n\n## Pixel-coordinate clicks\n\nThe pixel path (`click({pid, x, y})`) is for surfaces the\naccessibility tree doesn't reach — canvases, video players, WebGL,\ncustom-drawn controls. Coords are **window-local screenshot pixels**\n(same space as the PNG `get_window_state` returns). Top-left origin,\ny-down. The driver handles screen-point conversion internally.\nPassing `window_id` alongside `x, y` is optional but recommended —\nit pins the coordinate conversion to the window whose screenshot\nproduced the pixel.\n\nPNGs returned by `get_window_state` are capped at **1568 px long-side\nby default** (`max_image_dimension` config), matching Anthropic's\nmultimodal-vision downsampling limit. The image the model reasons\nover and the image the click tool's coordinate system lives in are\nthe **same resolution** — just look at the PNG, pick a pixel, click\nat that pixel. No scaling math.\n\nThis is the default because the mismatch between \"rendered\nthumbnail\" and \"native PNG\" was a recurring coord-estimation\nfootgun. If you opt out (explicit `max_image_dimension=0` for\npixel-perfect verification flows), the old rule applies: don't\neyeball coords from whatever your client renders — it may be\n2-4× smaller than the PNG on disk, and a 2% error in thumbnail\nspace becomes ~80 px in the real image.\n\nFor precise targeting on small \u002F dense UIs:\n\n1. `get_window_state({pid, window_id})` → image capped at 1568\n   long-side plus `screenshot_width` \u002F `screenshot_height`. Write to\n   disk via `--screenshot-out-file \u003Cpath>`.\n2. Look at the PNG. Since it matches what you see, pick the target\n   pixel directly.\n3. When precision matters, draw a crosshair on the image (do\n   **not** crop — cropping loses the coordinate system) and verify\n   before clicking:\n\n```python\nfrom PIL import Image, ImageDraw\nimg = Image.open('\u002Ftmp\u002Fshot.png')\ndraw = ImageDraw.Draw(img)\nx, y = \u003Cyour_coordinate>\nr = 18\ndraw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)\ndraw.line([x-30, y, x+30, y], fill='red', width=3)\ndraw.line([x, y-30, x, y+30], fill='red', width=3)\nimg.save('\u002Ftmp\u002Fshot_annotated.png')\n```\n\n4. Only dispatch the click after the user (or your own re-read of\n   the annotated image) confirms the crosshair is on target.\n\nAddressing variants:\n\n- `click({pid, x, y})` — single left-click.\n- `click({pid, x, y, count: 2})` — double-click.\n- `click({pid, x, y, modifier: [\"cmd\"\\|\"ctrl\"]})` — modifier click.\n  Accepts any subset of `cmd\u002Fshift\u002Foption\u002Falt\u002Fctrl`.\n- `right_click({pid, x, y})` — also takes `modifier`.\n\nThe pixel path animates the agent cursor overlay but never warps\nthe real cursor (the per-pid event paths the driver uses on macOS\nand Windows route around HID synthesis). If the pid has no on-screen\nwindow the call errors with `pid X has no on-screen window` — you\nneed a visible window to anchor the conversion. Dispatch details\n(SkyLight on macOS, layered UIA+PostMessage on Windows) are in the\nper-OS companion files.\n\n## Web-rendered apps (browsers, Electron, Tauri)\n\nFor Chrome \u002F Edge \u002F Brave \u002F Arc \u002F Safari, Electron apps (Slack,\nVSCode, Notion, Discord), and Tauri apps — see **`WEB_APPS.md`**.\n\nCovers: sparse accessibility tree population (retry-once pattern for\nChromium), URL navigation via omnibox suggestions, the `set_value`\nworkaround for keyboard commits on **minimized** windows (Return\nsilently no-ops — symptom is a system bell; use `set_value` or click\na clickable equivalent), scrolling via synthetic PageUp\u002FDown\nkeystrokes, in-page clicks, and typing into web inputs.\n\nBrowser JS primitives are now **cross-platform** via the `page` tool —\nmacOS uses Apple Events for Chrome\u002FBrave\u002FEdge\u002FSafari + CDP for Electron\n(see `MACOS.md`); Windows + Linux use UIA \u002F AT-SPI for `get_text` \u002F\n`query_dom` and the shared CDP client for `execute_javascript` (browser\nmust be launched with `--remote-debugging-port=N` and the port exported\nas `CUA_DRIVER_CDP_PORT`).\n\n## Re-snapshot and verify — mandatory\n\n**Always** call `get_window_state({pid, window_id})` after the\naction. This isn't optional verification — it's the second half of\nthe snapshot invariant.\n\nCheck the tree diff: a changed value, a new element, a new window,\nor the disappearance of the thing you just clicked (menus collapse\nafter selection, buttons may become disabled, etc.). The re-snapshot\ngives you both the tree and the screenshot, so you cross-check the tree\ndiff against the pixels in one call — and when you're only confirming a\ntree change, `include_screenshot:false` skips the grab.\n\nSwitch to an **element px action** only on a real signal: the action\nresponse carried `effect:\"suspected_noop\"`, the re-snapshot came back\n`degraded` (empty tree → non-AX surface), the tree looks\nunchanged\u002Funreadable or disagrees with the screenshot, or\n`escalation.recommended` points you there (`px`). That's the\nverify-then-escalate ladder in the behavior-matrix section. If the tree\nis unchanged AND the screenshot confirms nothing moved, the action\nlikely failed silently — **tell the user what you attempted and what\nyou observed**, don't paper over with \"done\" language (and consider\n`delivery_mode:\"foreground\"` when `escalation.recommended ==\n\"foreground\"`). Agents that skip this step report success on\nsilently-dropped actions — the single most common failure mode.\n\n## Recording trajectories\n\nSession-scoped action recording + replay, for demos, regressions,\nand training data. Only invoke when the user explicitly asks to\nrecord a session — the skill does not auto-enable this. CLI surface:\n`cua-driver recording start|stop|status`; raw tools:\n`start_recording` \u002F `stop_recording`. Video capture (main display →\n`recording.mp4`) is on by default; pass `record_video: false` to opt out.\n\nSee **`RECORDING.md`** for the full flow: enable\u002Fdisable, turn folder\ncontents, replay via `replay_trajectory`, and the element_index\ndoesn't-survive-across-sessions caveat.\n\n## Common error patterns (cross-platform)\n\n| Error text | Meaning | Fix |\n|---|---|---|\n| `No cached AX state for pid X window_id W` | You either skipped `get_window_state` this turn, or passed a different `window_id` to the click than the one the snapshot cached against | Call `get_window_state({pid: X, window_id: W})` first — the same window_id you intend to click in |\n| `Invalid element_index N for pid X window_id W` | Index is stale or out of range | Re-run `get_window_state` with the same window_id, pick a fresh index from the new tree |\n| `window_id W belongs to pid P, not …` | Passed a window_id that's owned by a different process | Use `list_windows({pid: X})` to enumerate this pid's own windows |\n| `AX action … failed with code …` \u002F `UIA invoke failed` | Element doesn't support the default action | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center |\n| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this *exact* string for BOTH a permission-prompt denial AND a manual interrupt (Esc \u002F stop) of a running tool — they are indistinguishable from the tool result | Treat as \"tool canceled, no result, await the user.\" Do NOT paraphrase (\"you stopped me\") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |\n\nPlatform-specific errors (TCC dialogs on macOS, Session 0 \u002F UAC\nprompts on Windows, AT-SPI bus issues on Linux) live in their\nrespective companion files.\n\n## Things to avoid\n\n- **Never** reuse an `element_index` across a re-snapshot of the same pid.\n- **Don't conflate the two addressing modes.** The tree gives you\n  `element_index` handles; the screenshot (same call) gives you the\n  pixel frame. An **element ax action** addresses by index, an\n  **element px action** by `x,y`. Default to `element_index` and only\n  do a px action on a real signal (`suspected_noop` \u002F `degraded` \u002F\n  repeated labels \u002F tree-disagrees-with-pixels). Don't pass an\n  `element_index` you read off the screenshot, and don't pixel-click a\n  coordinate you computed from the tree's (possibly lying) frame\n  without checking it against the image.\n- **Prefer accessibility actions over pixels.** `click({pid, x, y})`\n  works for canvas \u002F WebView regions, but it lands blindly on raw\n  coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,\n  toolbar items, keyboard shortcuts) before dropping to coordinates.\n  (The AX path does **not** skip the agent-cursor overlay — it seeds and\n  pulses the session cursor and draws a focus rect on the targeted\n  element; it just doesn't play a long glide on the very first action.\n  See \"Agent cursor overlay\" for the demo-recording caveat.)\n- **Never** drive destructive actions (delete files, close unsaved\n  documents, send messages, submit forms) without explicit user\n  intent for that specific destructive step.\n- **Never** launch apps autonomously; confirm with the user first\n  unless their original request clearly implies the launch.\n\n## Example end-to-end task\n\n**User:** \"Open the Downloads folder in the system file manager.\"\n\n1. `launch_app({bundle_id: \"com.apple.finder\", urls: [\"~\u002FDownloads\"]})`\n   on macOS, or `launch_app({name: \"explorer\", args: [\"%USERPROFILE%\\\\Downloads\"]})`\n   on Windows. Returns `{pid, windows: [{window_id, title, ...}]}`.\n   Idempotent launch; the driver opens a hidden window via the\n   platform's launch primitive — zero activation, no focus steal.\n2. `get_window_state({pid, window_id})` → verify the expected window\n   title is present with a populated tree (sidebar, list view, files).\n3. Done.\n\nPlatform-specific examples and edge cases (Finder menu navigation,\nExplorer ribbon, GNOME Files) live in the per-OS companion files.\n",{"data":38,"body":39},{"name":4,"description":6},{"type":40,"children":41},"root",[42,49,64,71,84,164,169,221,234,240,250,335,340,346,387,415,422,434,478,538,559,565,593,705,710,832,838,896,922,969,988,994,1010,1055,1060,1066,1114,1120,1134,1158,1221,1233,1305,1392,1410,1422,1509,1559,1657,1734,1804,1812,1825,1994,2031,2106,2112,2124,2134,2180,2188,2385,2402,2408,2417,2455,2511,2575,2651,2697,2703,2766,2771,2920,2928,3019,3045,3061,3125,3177,3182,3216,3222,3255,3756,3772,3789,3795,3927,3932,4134,4208,4229,4235,4276,4309,4322,4327,4378,4462,4470,4475,4534,4547,4553,4567,4592,4657,4663,4680,4692,4754,4760,4803,4825,4831,5047,5052,5058,5181,5187,5197,5242,5247],{"type":43,"tag":44,"props":45,"children":46},"element","h1",{"id":4},[47],{"type":48,"value":4},"text",{"type":43,"tag":50,"props":51,"children":52},"p",{},[53,55,62],{"type":48,"value":54},"Orchestrates cross-platform app automation via ",{"type":43,"tag":56,"props":57,"children":59},"code",{"className":58},[],[60],{"type":48,"value":61},"cua-driver",{"type":48,"value":63},". Whenever\na user asks to drive a native app, follow the loop in this skill\nrather than calling tools ad-hoc — the snapshot-before-action\ninvariant is not optional and silently breaks if you skip it.",{"type":43,"tag":65,"props":66,"children":68},"h2",{"id":67},"platform-specific-reading-read-this-first",[69],{"type":48,"value":70},"Platform-specific reading — read this first",{"type":43,"tag":50,"props":72,"children":73},{},[74,76,82],{"type":48,"value":75},"This file is the ",{"type":43,"tag":77,"props":78,"children":79},"strong",{},[80],{"type":48,"value":81},"cross-platform core",{"type":48,"value":83},": snapshot invariant, CLI vs\nMCP choice, tool surface naming, behavior matrix, canonical loop,\npixel-click contract, common failure modes. The platform-specific\nmaterial (forbidden-list, accessibility tree implementation, launch\nsemantics, click dispatch) lives in companion files in this same\ndirectory:",{"type":43,"tag":85,"props":86,"children":87},"ul",{},[88,130,147],{"type":43,"tag":89,"props":90,"children":91},"li",{},[92,97,99,105,107,113,115,121,122,128],{"type":43,"tag":77,"props":93,"children":94},{},[95],{"type":48,"value":96},"macOS",{"type":48,"value":98}," — read ",{"type":43,"tag":56,"props":100,"children":102},{"className":101},[],[103],{"type":48,"value":104},"MACOS.md",{"type":48,"value":106}," (no-foreground contract, forbidden\n",{"type":43,"tag":56,"props":108,"children":110},{"className":109},[],[111],{"type":48,"value":112},"open",{"type":48,"value":114},"\u002F",{"type":43,"tag":56,"props":116,"children":118},{"className":117},[],[119],{"type":48,"value":120},"osascript",{"type":48,"value":114},{"type":43,"tag":56,"props":123,"children":125},{"className":124},[],[126],{"type":48,"value":127},"cliclick",{"type":48,"value":129}," invocations, AXMenuBar navigation,\nSkyLight pixel-click dispatch, Apple-Events JS bridge).",{"type":43,"tag":89,"props":131,"children":132},{},[133,138,139,145],{"type":43,"tag":77,"props":134,"children":135},{},[136],{"type":48,"value":137},"Windows",{"type":48,"value":98},{"type":43,"tag":56,"props":140,"children":142},{"className":141},[],[143],{"type":48,"value":144},"WINDOWS.md",{"type":48,"value":146}," (UIA tree vs AX, UWP \u002F\nApplicationFrameHost hosting, layered UIA+PostMessage click chain,\nSession 0 isolation, Windows-specific focus-steal vectors).",{"type":43,"tag":89,"props":148,"children":149},{},[150,155,156,162],{"type":43,"tag":77,"props":151,"children":152},{},[153],{"type":48,"value":154},"Linux",{"type":48,"value":98},{"type":43,"tag":56,"props":157,"children":159},{"className":158},[],[160],{"type":48,"value":161},"LINUX.md",{"type":48,"value":163}," (X11 background input via AT-SPI +\nXSendEvent, recording, Wayland opt-in\u002Fpreview).",{"type":43,"tag":50,"props":165,"children":166},{},[167],{"type":48,"value":168},"Cross-cutting topics also have their own files:",{"type":43,"tag":85,"props":170,"children":171},{},[172,191,210],{"type":43,"tag":89,"props":173,"children":174},{},[175,181,183,189],{"type":43,"tag":56,"props":176,"children":178},{"className":177},[],[179],{"type":48,"value":180},"WEB_APPS.md",{"type":48,"value":182}," — browser \u002F Electron \u002F Tauri specifics (sparse AX\ntrees, omnibox navigation, the ",{"type":43,"tag":56,"props":184,"children":186},{"className":185},[],[187],{"type":48,"value":188},"set_value",{"type":48,"value":190}," workaround for\nminimized windows, tabs-vs-windows guidance).",{"type":43,"tag":89,"props":192,"children":193},{},[194,200,202,208],{"type":43,"tag":56,"props":195,"children":197},{"className":196},[],[198],{"type":48,"value":199},"RECORDING.md",{"type":48,"value":201}," — session recording + ",{"type":43,"tag":56,"props":203,"children":205},{"className":204},[],[206],{"type":48,"value":207},"replay_trajectory",{"type":48,"value":209},".",{"type":43,"tag":89,"props":211,"children":212},{},[213,219],{"type":43,"tag":56,"props":214,"children":216},{"className":215},[],[217],{"type":48,"value":218},"TESTS.md",{"type":48,"value":220}," — internal test surface.",{"type":43,"tag":50,"props":222,"children":223},{},[224,226,232],{"type":48,"value":225},"Use whichever combination matches the host. When in doubt, run\n",{"type":43,"tag":56,"props":227,"children":229},{"className":228},[],[230],{"type":48,"value":231},"cua-driver doctor",{"type":48,"value":233}," — it reports the platform and the right entry\npoint.",{"type":43,"tag":65,"props":235,"children":237},{"id":236},"the-no-foreground-principle-cross-platform",[238],{"type":48,"value":239},"The no-foreground principle (cross-platform)",{"type":43,"tag":50,"props":241,"children":242},{},[243,248],{"type":43,"tag":77,"props":244,"children":245},{},[246],{"type":48,"value":247},"The user's frontmost app MUST NOT change.",{"type":48,"value":249}," Every platform has its\nown list of forbidden commands; the principle is universal:",{"type":43,"tag":85,"props":251,"children":252},{},[253,293],{"type":43,"tag":89,"props":254,"children":255},{},[256,258,263,265,270,272,277,279,285,287,292],{"type":48,"value":257},"macOS: any ",{"type":43,"tag":56,"props":259,"children":261},{"className":260},[],[262],{"type":48,"value":112},{"type":48,"value":264}," invocation, any ",{"type":43,"tag":56,"props":266,"children":268},{"className":267},[],[269],{"type":48,"value":120},{"type":48,"value":271}," that mutates GUI\nstate, ",{"type":43,"tag":56,"props":273,"children":275},{"className":274},[],[276],{"type":48,"value":127},{"type":48,"value":278},", ",{"type":43,"tag":56,"props":280,"children":282},{"className":281},[],[283],{"type":48,"value":284},"cghidEventTap",{"type":48,"value":286}," writes targeting another app's\nwindow. Full list in ",{"type":43,"tag":56,"props":288,"children":290},{"className":289},[],[291],{"type":48,"value":104},{"type":48,"value":209},{"type":43,"tag":89,"props":294,"children":295},{},[296,298,304,306,312,313,319,321,327,329,334],{"type":48,"value":297},"Windows: any ",{"type":43,"tag":56,"props":299,"children":301},{"className":300},[],[302],{"type":48,"value":303},"Start-Process",{"type":48,"value":305}," that triggers a ",{"type":43,"tag":56,"props":307,"children":309},{"className":308},[],[310],{"type":48,"value":311},"ShowWindow",{"type":48,"value":114},{"type":43,"tag":56,"props":314,"children":316},{"className":315},[],[317],{"type":48,"value":318},"SetForegroundWindow",{"type":48,"value":320},"\non the target, ",{"type":43,"tag":56,"props":322,"children":324},{"className":323},[],[325],{"type":48,"value":326},"WScript.Shell.AppActivate",{"type":48,"value":328},", attaching to the\nforeground thread for input forwarding. Full list in ",{"type":43,"tag":56,"props":330,"children":332},{"className":331},[],[333],{"type":48,"value":144},{"type":48,"value":209},{"type":43,"tag":50,"props":336,"children":337},{},[338],{"type":48,"value":339},"If you reach for a command that says \"activate\", \"foreground\",\n\"raise\", or \"make key\", stop and translate to the cua-driver tool\nthat does the same intent without focus-stealing.",{"type":43,"tag":65,"props":341,"children":343},{"id":342},"defaults-always-prefer-cua-driver-over-shell-shims",[344],{"type":48,"value":345},"Defaults — always prefer cua-driver over shell shims",{"type":43,"tag":50,"props":347,"children":348},{},[349,361,363,369,371,377,379,385],{"type":43,"tag":77,"props":350,"children":351},{},[352,354,359],{"type":48,"value":353},"Default transport is the ",{"type":43,"tag":56,"props":355,"children":357},{"className":356},[],[358],{"type":48,"value":61},{"type":48,"value":360}," CLI",{"type":48,"value":362}," — ",{"type":43,"tag":56,"props":364,"children":366},{"className":365},[],[367],{"type":48,"value":368},"Bash",{"type":48,"value":370}," shelling out\nto ",{"type":43,"tag":56,"props":372,"children":374},{"className":373},[],[375],{"type":48,"value":376},"cua-driver \u003Ctool-name> '\u003CJSON-args>'",{"type":48,"value":378},". MCP tools (prefix\n",{"type":43,"tag":56,"props":380,"children":382},{"className":381},[],[383],{"type":48,"value":384},"mcp__cua-driver__*",{"type":48,"value":386},") only when the user explicitly asks for them.\nCLI wins because it picks up rebuilds instantly, failures are\neasier to diagnose, and there's no per-tool schema-load overhead.",{"type":43,"tag":50,"props":388,"children":389},{},[390,392,398,399,405,407,413],{"type":48,"value":391},"Every reference to ",{"type":43,"tag":56,"props":393,"children":395},{"className":394},[],[396],{"type":48,"value":397},"click(...)",{"type":48,"value":278},{"type":43,"tag":56,"props":400,"children":402},{"className":401},[],[403],{"type":48,"value":404},"get_window_state(...)",{"type":48,"value":406}," etc. in this\nskill means ",{"type":43,"tag":56,"props":408,"children":410},{"className":409},[],[411],{"type":48,"value":412},"cua-driver click '{...}'",{"type":48,"value":414}," — translate to MCP form only\nwhen MCP is requested.",{"type":43,"tag":416,"props":417,"children":419},"h3",{"id":418},"claude-code-computer-use-compatibility-mode",[420],{"type":48,"value":421},"Claude Code computer-use compatibility mode",{"type":43,"tag":50,"props":423,"children":424},{},[425,427,432],{"type":48,"value":426},"For normal Claude Code use, keep the default CLI or ",{"type":43,"tag":56,"props":428,"children":430},{"className":429},[],[431],{"type":48,"value":61},{"type":48,"value":433}," MCP\nserver path above. If the user explicitly wants Claude Code's\nvision\u002Fcomputer-use-style flow, they can register:",{"type":43,"tag":435,"props":436,"children":441},"pre",{"className":437,"code":438,"language":439,"meta":440,"style":440},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","cua-driver mcp-config --client claude   # then paste + run the printed line\n","bash","",[442],{"type":43,"tag":56,"props":443,"children":444},{"__ignoreMap":440},[445],{"type":43,"tag":446,"props":447,"children":450},"span",{"class":448,"line":449},"line",1,[451,456,462,467,472],{"type":43,"tag":446,"props":452,"children":454},{"style":453},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[455],{"type":48,"value":61},{"type":43,"tag":446,"props":457,"children":459},{"style":458},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[460],{"type":48,"value":461}," mcp-config",{"type":43,"tag":446,"props":463,"children":464},{"style":458},[465],{"type":48,"value":466}," --client",{"type":43,"tag":446,"props":468,"children":469},{"style":458},[470],{"type":48,"value":471}," claude",{"type":43,"tag":446,"props":473,"children":475},{"style":474},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[476],{"type":48,"value":477},"   # then paste + run the printed line\n",{"type":43,"tag":50,"props":479,"children":480},{},[481,483,489,491,496,498,504,506,512,514,520,522,528,530,536],{"type":48,"value":482},"Observation: Claude Code vision flows appear to treat a screenshot\nMCP tool as the image-grounding anchor. This compatibility mode keeps\nthe normal CuaDriver tools and changes only ",{"type":43,"tag":56,"props":484,"children":486},{"className":485},[],[487],{"type":48,"value":488},"screenshot",{"type":48,"value":490},". The\ncompatibility ",{"type":43,"tag":56,"props":492,"children":494},{"className":493},[],[495],{"type":48,"value":488},{"type":48,"value":497}," requires ",{"type":43,"tag":56,"props":499,"children":501},{"className":500},[],[502],{"type":48,"value":503},"pid",{"type":48,"value":505}," and ",{"type":43,"tag":56,"props":507,"children":509},{"className":508},[],[510],{"type":48,"value":511},"window_id",{"type":48,"value":513},", captures\nonly that target window, and returns the window-local pixel\ncoordinate frame. Start with ",{"type":43,"tag":56,"props":515,"children":517},{"className":516},[],[518],{"type":48,"value":519},"launch_app",{"type":48,"value":521}," or ",{"type":43,"tag":56,"props":523,"children":525},{"className":524},[],[526],{"type":48,"value":527},"list_windows",{"type":48,"value":529},", then\ncall ",{"type":43,"tag":56,"props":531,"children":533},{"className":532},[],[534],{"type":48,"value":535},"screenshot({pid, window_id})",{"type":48,"value":537},"; do not assume desktop\ncoordinates or a full-screen capture.",{"type":43,"tag":50,"props":539,"children":540},{},[541,543,549,551,557],{"type":48,"value":542},"Use MCP for this Claude Code vision\u002Fcomputer-use-style path. Do not\nshell out to ",{"type":43,"tag":56,"props":544,"children":546},{"className":545},[],[547],{"type":48,"value":548},"cua-driver screenshot",{"type":48,"value":550}," as a substitute: CLI screenshots\nstill work as CuaDriver calls, but they do not expose the\n",{"type":43,"tag":56,"props":552,"children":554},{"className":553},[],[555],{"type":48,"value":556},"mcp__cua-computer-use__screenshot",{"type":48,"value":558}," tool name that Claude Code\nappears to use as the image-grounding cue.",{"type":43,"tag":65,"props":560,"children":562},{"id":561},"using-cua-driver-from-the-shell",[563],{"type":48,"value":564},"Using cua-driver from the shell",{"type":43,"tag":50,"props":566,"children":567},{},[568,570,576,578,584,586,591],{"type":48,"value":569},"Tool names are ",{"type":43,"tag":56,"props":571,"children":573},{"className":572},[],[574],{"type":48,"value":575},"snake_case",{"type":48,"value":577},", management subcommands are\n",{"type":43,"tag":56,"props":579,"children":581},{"className":580},[],[582],{"type":48,"value":583},"kebab-case",{"type":48,"value":585}," — no ambiguity. Tools invoked as ",{"type":43,"tag":56,"props":587,"children":589},{"className":588},[],[590],{"type":48,"value":376},{"type":48,"value":592},". Management subcommands:",{"type":43,"tag":85,"props":594,"children":595},{},[596,629,646,662,678],{"type":43,"tag":89,"props":597,"children":598},{},[599,605,607,612,614,620,622,627],{"type":43,"tag":56,"props":600,"children":602},{"className":601},[],[603],{"type":48,"value":604},"cua-driver serve",{"type":48,"value":606}," — start persistent daemon (",{"type":43,"tag":77,"props":608,"children":609},{},[610],{"type":48,"value":611},"required",{"type":48,"value":613}," for\n",{"type":43,"tag":56,"props":615,"children":617},{"className":616},[],[618],{"type":48,"value":619},"element_index",{"type":48,"value":621}," workflows; without it each CLI invocation spawns a\nfresh process and the per-pid element cache dies between calls).\nmacOS users: see ",{"type":43,"tag":56,"props":623,"children":625},{"className":624},[],[626],{"type":48,"value":104},{"type":48,"value":628}," for the LaunchServices-routed launch\nform.",{"type":43,"tag":89,"props":630,"children":631},{},[632,638,640],{"type":43,"tag":56,"props":633,"children":635},{"className":634},[],[636],{"type":48,"value":637},"cua-driver stop",{"type":48,"value":639}," \u002F ",{"type":43,"tag":56,"props":641,"children":643},{"className":642},[],[644],{"type":48,"value":645},"status",{"type":43,"tag":89,"props":647,"children":648},{},[649,655,656],{"type":43,"tag":56,"props":650,"children":652},{"className":651},[],[653],{"type":48,"value":654},"cua-driver list-tools",{"type":48,"value":278},{"type":43,"tag":56,"props":657,"children":659},{"className":658},[],[660],{"type":48,"value":661},"describe \u003Ctool>",{"type":43,"tag":89,"props":663,"children":664},{},[665,671,673],{"type":43,"tag":56,"props":666,"children":668},{"className":667},[],[669],{"type":48,"value":670},"cua-driver recording start|stop|status",{"type":48,"value":672}," — see ",{"type":43,"tag":56,"props":674,"children":676},{"className":675},[],[677],{"type":48,"value":199},{"type":43,"tag":89,"props":679,"children":680},{},[681,687,689,695,697,703],{"type":43,"tag":56,"props":682,"children":684},{"className":683},[],[685],{"type":48,"value":686},"cua-driver check-update [--json] [--no-cache]",{"type":48,"value":688}," — read-only \"is a newer release available?\" probe. Same payload as the ",{"type":43,"tag":56,"props":690,"children":692},{"className":691},[],[693],{"type":48,"value":694},"check_for_update",{"type":48,"value":696}," MCP tool; pair with ",{"type":43,"tag":56,"props":698,"children":700},{"className":699},[],[701],{"type":48,"value":702},"cua-driver update --apply",{"type":48,"value":704}," to install.",{"type":43,"tag":50,"props":706,"children":707},{},[708],{"type":48,"value":709},"Canonical multi-step workflow (example shape — platform-specific\nlaunch idioms in the per-OS companion file):",{"type":43,"tag":435,"props":711,"children":713},{"className":437,"code":712,"language":439,"meta":440,"style":440},"cua-driver serve\ncua-driver launch_app '{\"bundle_id\":\"...\"}'\n# → {pid: 844, windows: [{window_id: 10725, ...}]}\ncua-driver get_window_state '{\"pid\":844,\"window_id\":10725}'\ncua-driver click '{\"pid\":844,\"window_id\":10725,\"element_index\":14}'\ncua-driver stop\n",[714],{"type":43,"tag":56,"props":715,"children":716},{"__ignoreMap":440},[717,729,758,767,793,819],{"type":43,"tag":446,"props":718,"children":719},{"class":448,"line":449},[720,724],{"type":43,"tag":446,"props":721,"children":722},{"style":453},[723],{"type":48,"value":61},{"type":43,"tag":446,"props":725,"children":726},{"style":458},[727],{"type":48,"value":728}," serve\n",{"type":43,"tag":446,"props":730,"children":732},{"class":448,"line":731},2,[733,737,742,748,753],{"type":43,"tag":446,"props":734,"children":735},{"style":453},[736],{"type":48,"value":61},{"type":43,"tag":446,"props":738,"children":739},{"style":458},[740],{"type":48,"value":741}," launch_app",{"type":43,"tag":446,"props":743,"children":745},{"style":744},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[746],{"type":48,"value":747}," '",{"type":43,"tag":446,"props":749,"children":750},{"style":458},[751],{"type":48,"value":752},"{\"bundle_id\":\"...\"}",{"type":43,"tag":446,"props":754,"children":755},{"style":744},[756],{"type":48,"value":757},"'\n",{"type":43,"tag":446,"props":759,"children":761},{"class":448,"line":760},3,[762],{"type":43,"tag":446,"props":763,"children":764},{"style":474},[765],{"type":48,"value":766},"# → {pid: 844, windows: [{window_id: 10725, ...}]}\n",{"type":43,"tag":446,"props":768,"children":770},{"class":448,"line":769},4,[771,775,780,784,789],{"type":43,"tag":446,"props":772,"children":773},{"style":453},[774],{"type":48,"value":61},{"type":43,"tag":446,"props":776,"children":777},{"style":458},[778],{"type":48,"value":779}," get_window_state",{"type":43,"tag":446,"props":781,"children":782},{"style":744},[783],{"type":48,"value":747},{"type":43,"tag":446,"props":785,"children":786},{"style":458},[787],{"type":48,"value":788},"{\"pid\":844,\"window_id\":10725}",{"type":43,"tag":446,"props":790,"children":791},{"style":744},[792],{"type":48,"value":757},{"type":43,"tag":446,"props":794,"children":796},{"class":448,"line":795},5,[797,801,806,810,815],{"type":43,"tag":446,"props":798,"children":799},{"style":453},[800],{"type":48,"value":61},{"type":43,"tag":446,"props":802,"children":803},{"style":458},[804],{"type":48,"value":805}," click",{"type":43,"tag":446,"props":807,"children":808},{"style":744},[809],{"type":48,"value":747},{"type":43,"tag":446,"props":811,"children":812},{"style":458},[813],{"type":48,"value":814},"{\"pid\":844,\"window_id\":10725,\"element_index\":14}",{"type":43,"tag":446,"props":816,"children":817},{"style":744},[818],{"type":48,"value":757},{"type":43,"tag":446,"props":820,"children":822},{"class":448,"line":821},6,[823,827],{"type":43,"tag":446,"props":824,"children":825},{"style":453},[826],{"type":48,"value":61},{"type":43,"tag":446,"props":828,"children":829},{"style":458},[830],{"type":48,"value":831}," stop\n",{"type":43,"tag":65,"props":833,"children":835},{"id":834},"agent-cursor-overlay",[836],{"type":48,"value":837},"Agent cursor overlay",{"type":43,"tag":50,"props":839,"children":840},{},[841,843,849,851,857,859,865,867,873,874,880,881,887,888,894],{"type":48,"value":842},"Visual cursor overlay for demos and screen recordings. Default:\nenabled — you do NOT need to enable it. Toggle with\n",{"type":43,"tag":56,"props":844,"children":846},{"className":845},[],[847],{"type":48,"value":848},"cua-driver set_agent_cursor_enabled '{\"enabled\":true|false}'",{"type":48,"value":850}," only to\nhide or re-show it. A triangle pointer Bezier-glides to each click\ntarget, ring-ripples on landing, idle-hides after ~1.5s. Motion knobs:\n",{"type":43,"tag":56,"props":852,"children":854},{"className":853},[],[855],{"type":48,"value":856},"set_agent_cursor_motion",{"type":48,"value":858}," takes any subset of ",{"type":43,"tag":56,"props":860,"children":862},{"className":861},[],[863],{"type":48,"value":864},"start_handle",{"type":48,"value":866},",\n",{"type":43,"tag":56,"props":868,"children":870},{"className":869},[],[871],{"type":48,"value":872},"end_handle",{"type":48,"value":278},{"type":43,"tag":56,"props":875,"children":877},{"className":876},[],[878],{"type":48,"value":879},"arc_size",{"type":48,"value":278},{"type":43,"tag":56,"props":882,"children":884},{"className":883},[],[885],{"type":48,"value":886},"arc_flow",{"type":48,"value":278},{"type":43,"tag":56,"props":889,"children":891},{"className":890},[],[892],{"type":48,"value":893},"spring",{"type":48,"value":895}," — tuneable at runtime,\npersisted to config.",{"type":43,"tag":50,"props":897,"children":898},{},[899,904,906,912,914,920],{"type":43,"tag":77,"props":900,"children":901},{},[902],{"type":48,"value":903},"Per-session cursors.",{"type":48,"value":905}," Each MCP session automatically owns its own\ncursor, keyed by the session's id (the proxy mints one session id per\nMCP connection and the daemon scopes the cursor, config overrides, and\nrecording to it). You normally pass nothing — the session key is wired\nthrough for you. Pass an explicit ",{"type":43,"tag":56,"props":907,"children":909},{"className":908},[],[910],{"type":48,"value":911},"cursor_id",{"type":48,"value":913}," only to ",{"type":43,"tag":915,"props":916,"children":917},"em",{},[918],{"type":48,"value":919},"deliberately\nshare",{"type":48,"value":921}," one cursor across sessions. When a session ends (the MCP client\ndisconnects) its cursor is removed automatically.",{"type":43,"tag":50,"props":923,"children":924},{},[925,930,932,937,939,944,946,951,953,959,961,967],{"type":43,"tag":77,"props":926,"children":927},{},[928],{"type":48,"value":929},"Visibility caveat (AX runs).",{"type":48,"value":931}," On a pure accessibility-action run\n(clicking by ",{"type":43,"tag":56,"props":933,"children":935},{"className":934},[],[936],{"type":48,"value":619},{"type":48,"value":938},"), the first action ",{"type":43,"tag":77,"props":940,"children":941},{},[942],{"type":48,"value":943},"seeds the cursor\non-screen a short distance from the target and plays a brief glide +\npulse",{"type":48,"value":945}," — not the long Bezier sweep a cursor already on-screen would\ntrace from its previous spot. It's subtle and easy to miss in a\nrecording. If you want a clearly ",{"type":43,"tag":915,"props":947,"children":948},{},[949],{"type":48,"value":950},"gliding",{"type":48,"value":952}," cursor for a demo or screen\nrecording, do a pixel click (",{"type":43,"tag":56,"props":954,"children":956},{"className":955},[],[957],{"type":48,"value":958},"click({pid,x,y})",{"type":48,"value":960},") or a ",{"type":43,"tag":56,"props":962,"children":964},{"className":963},[],[965],{"type":48,"value":966},"move_agent_cursor",{"type":48,"value":968},"\nfirst to put the cursor on-screen; subsequent AX actions then glide the\nfull path normally.",{"type":43,"tag":50,"props":970,"children":971},{},[972,974,979,981,986],{"type":48,"value":973},"Requires the daemon process's UI runloop, which ",{"type":43,"tag":56,"props":975,"children":977},{"className":976},[],[978],{"type":48,"value":604},{"type":48,"value":980}," \u002F\n",{"type":43,"tag":56,"props":982,"children":984},{"className":983},[],[985],{"type":48,"value":19},{"type":48,"value":987}," bootstraps. One-shot CLI invocations skip the overlay entirely.",{"type":43,"tag":65,"props":989,"children":991},{"id":990},"the-core-invariant-snapshot-before-and-after-every-action",[992],{"type":48,"value":993},"The core invariant — snapshot before AND after every action",{"type":43,"tag":50,"props":995,"children":996},{},[997,1008],{"type":43,"tag":77,"props":998,"children":999},{},[1000,1002],{"type":48,"value":1001},"Every action MUST be bracketed by ",{"type":43,"tag":56,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":48,"value":1007},"get_window_state(pid, window_id)",{"type":48,"value":1009},":",{"type":43,"tag":85,"props":1011,"children":1012},{},[1013,1045],{"type":43,"tag":89,"props":1014,"children":1015},{},[1016,1021,1023,1028,1030,1036,1038,1044],{"type":43,"tag":77,"props":1017,"children":1018},{},[1019],{"type":48,"value":1020},"Before",{"type":48,"value":1022}," — the pre-action snapshot resolves the ",{"type":43,"tag":56,"props":1024,"children":1026},{"className":1025},[],[1027],{"type":48,"value":619},{"type":48,"value":1029},"\nyou're about to use. Indices from previous turns are stale; the\nserver replaces the element index map on every snapshot, keyed\non ",{"type":43,"tag":56,"props":1031,"children":1033},{"className":1032},[],[1034],{"type":48,"value":1035},"(pid, window_id)",{"type":48,"value":1037},". Indices from turn N don't resolve in turn\nN+1, and indices from window A don't resolve against window B of\nthe same app. Skip this and element-indexed actions fail with\n",{"type":43,"tag":56,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":48,"value":1043},"No cached AX state",{"type":48,"value":209},{"type":43,"tag":89,"props":1046,"children":1047},{},[1048,1053],{"type":43,"tag":77,"props":1049,"children":1050},{},[1051],{"type":48,"value":1052},"After",{"type":48,"value":1054}," — the post-action snapshot verifies the action actually\nlanded. Without it you can't tell a silent no-op from a real\neffect. The accessibility-tree change (new value, new window,\ndisappeared menu, disabled button, etc.) is your evidence that\nthe action fired. If nothing changed, the action probably failed\nsilently — say so, don't assume success.",{"type":43,"tag":50,"props":1056,"children":1057},{},[1058],{"type":48,"value":1059},"This applies to pixel clicks too — re-snapshot after to confirm the\nclick landed on the intended target.",{"type":43,"tag":416,"props":1061,"children":1063},{"id":1062},"why-window-selection-is-the-callers-job-now",[1064],{"type":48,"value":1065},"Why window selection is the caller's job now",{"type":43,"tag":50,"props":1067,"children":1068},{},[1069,1075,1077,1083,1085,1090,1092,1097,1099,1105,1107,1112],{"type":43,"tag":56,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":48,"value":1074},"get_app_state",{"type":48,"value":1076}," used to pick a window for you via a max-area heuristic\nthat returned the wrong surface on apps with large off-screen utility\npanels. Concrete reproducer: IINA's OpenSubtitles helper (600×432\noff-screen) out-area'd the visible 320×240 player window, so\n",{"type":43,"tag":56,"props":1078,"children":1080},{"className":1079},[],[1081],{"type":48,"value":1082},"get_app_state(pid)",{"type":48,"value":1084}," screenshot'd the invisible panel and clicks landed\nthere silently. The new ",{"type":43,"tag":56,"props":1086,"children":1088},{"className":1087},[],[1089],{"type":48,"value":1007},{"type":48,"value":1091}," makes the\ncaller name the window explicitly — the driver validates that the\nwindow belongs to the pid and is on the current Space\u002Fdesktop, then\nsnapshots exactly what was asked for. Enumerate candidates via\n",{"type":43,"tag":56,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":48,"value":527},{"type":48,"value":1098}," or read the ",{"type":43,"tag":56,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":48,"value":1104},"windows",{"type":48,"value":1106}," array ",{"type":43,"tag":56,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":48,"value":519},{"type":48,"value":1113}," already\nreturns.",{"type":43,"tag":65,"props":1115,"children":1117},{"id":1116},"behavior-matrix",[1118],{"type":48,"value":1119},"Behavior matrix",{"type":43,"tag":416,"props":1121,"children":1123},{"id":1122},"perception-is-mode-agnostic-get_window_state-returns-both",[1124,1126,1132],{"type":48,"value":1125},"Perception is mode-agnostic — ",{"type":43,"tag":56,"props":1127,"children":1129},{"className":1128},[],[1130],{"type":48,"value":1131},"get_window_state",{"type":48,"value":1133}," returns BOTH",{"type":43,"tag":50,"props":1135,"children":1136},{},[1137,1142,1144,1149,1151,1156],{"type":43,"tag":56,"props":1138,"children":1140},{"className":1139},[],[1141],{"type":48,"value":1007},{"type":48,"value":1143}," ",{"type":43,"tag":77,"props":1145,"children":1146},{},[1147],{"type":48,"value":1148},"returns both the accessibility\ntree AND a screenshot by default.",{"type":48,"value":1150}," There is no capture mode to pick\nand nothing to configure — you ground on the tree and the screenshot\ntogether, and you cross-check one against the other. This matters\nbecause the tree ",{"type":43,"tag":77,"props":1152,"children":1153},{},[1154],{"type":48,"value":1155},"lies",{"type":48,"value":1157}," on some surfaces:",{"type":43,"tag":85,"props":1159,"children":1160},{},[1161,1185,1203],{"type":43,"tag":89,"props":1162,"children":1163},{},[1164,1169,1171,1176,1177,1183],{"type":43,"tag":77,"props":1165,"children":1166},{},[1167],{"type":48,"value":1168},"Electron",{"type":48,"value":1170}," echo-confirms a ",{"type":43,"tag":56,"props":1172,"children":1174},{"className":1173},[],[1175],{"type":48,"value":188},{"type":48,"value":639},{"type":43,"tag":56,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":48,"value":1182},"type_text",{"type":48,"value":1184}," against the AX\nshim while the rendered text view never changed.",{"type":43,"tag":89,"props":1186,"children":1187},{},[1188,1193,1195,1201],{"type":43,"tag":77,"props":1189,"children":1190},{},[1191],{"type":48,"value":1192},"Catalyst",{"type":48,"value":1194}," (iOSAppOnMac) exposes null \u002F placeholder ",{"type":43,"tag":56,"props":1196,"children":1198},{"className":1197},[],[1199],{"type":48,"value":1200},"AXValue",{"type":48,"value":1202},"s.",{"type":43,"tag":89,"props":1204,"children":1205},{},[1206,1211,1213,1219],{"type":43,"tag":77,"props":1207,"children":1208},{},[1209],{"type":48,"value":1210},"Virtualized \u002F off-viewport list rows",{"type":48,"value":1212}," report bogus frames (an\n",{"type":43,"tag":56,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":48,"value":1218},"h:1",{"type":48,"value":1220}," height, an off-screen origin) for rows that aren't actually\nlaid out.",{"type":43,"tag":50,"props":1222,"children":1223},{},[1224,1226,1231],{"type":48,"value":1225},"A grounding screenshot is present by default, so when the tree looks\nwrong you look at the pixels ",{"type":43,"tag":77,"props":1227,"children":1228},{},[1229],{"type":48,"value":1230},"in the same response",{"type":48,"value":1232}," — no second\ncapture, no mode flip.",{"type":43,"tag":1234,"props":1235,"children":1236},"blockquote",{},[1237],{"type":43,"tag":50,"props":1238,"children":1239},{},[1240,1252,1253,1258,1260,1266,1268,1273,1275,1281,1283,1288,1290,1296,1297,1303],{"type":43,"tag":77,"props":1241,"children":1242},{},[1243,1245,1251],{"type":48,"value":1244},"Perf opt-out — ",{"type":43,"tag":56,"props":1246,"children":1248},{"className":1247},[],[1249],{"type":48,"value":1250},"include_screenshot",{"type":48,"value":209},{"type":48,"value":1143},{"type":43,"tag":56,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":48,"value":1250},{"type":48,"value":1259},"\n(boolean, default ",{"type":43,"tag":56,"props":1261,"children":1263},{"className":1262},[],[1264],{"type":48,"value":1265},"true",{"type":48,"value":1267},") is the one knob, and it is a ",{"type":43,"tag":77,"props":1269,"children":1270},{},[1271],{"type":48,"value":1272},"perf",{"type":48,"value":1274}," knob,\nnot a modality choice. Default returns both (grounding-first). Pass\n",{"type":43,"tag":56,"props":1276,"children":1278},{"className":1277},[],[1279],{"type":48,"value":1280},"include_screenshot:false",{"type":48,"value":1282}," to skip the screen grab and get the tree\nonly — the cheap path when you're just ",{"type":43,"tag":77,"props":1284,"children":1285},{},[1286],{"type":48,"value":1287},"re-indexing before an\nelement ax action",{"type":48,"value":1289}," and don't need to re-ground on pixels. The\n",{"type":43,"tag":56,"props":1291,"children":1293},{"className":1292},[],[1294],{"type":48,"value":1295},"ax",{"type":48,"value":114},{"type":43,"tag":56,"props":1298,"children":1300},{"className":1299},[],[1301],{"type":48,"value":1302},"px",{"type":48,"value":1304}," decision still lives at action time, not here.",{"type":43,"tag":1234,"props":1306,"children":1307},{},[1308],{"type":43,"tag":50,"props":1309,"children":1310},{},[1311,1322,1324,1329,1331,1336,1338,1343,1345,1350,1351,1357,1358,1364,1366,1371,1372,1377,1378,1383,1385,1390],{"type":43,"tag":77,"props":1312,"children":1313},{},[1314,1320],{"type":43,"tag":56,"props":1315,"children":1317},{"className":1316},[],[1318],{"type":48,"value":1319},"capture_mode",{"type":48,"value":1321}," is DEPRECATED and ignored.",{"type":48,"value":1323}," It is still ",{"type":43,"tag":915,"props":1325,"children":1326},{},[1327],{"type":48,"value":1328},"accepted",{"type":48,"value":1330},"\non ",{"type":43,"tag":56,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":48,"value":1131},{"type":48,"value":1337}," so old callers don't error, but it has ",{"type":43,"tag":77,"props":1339,"children":1340},{},[1341],{"type":48,"value":1342},"no\neffect",{"type":48,"value":1344}," — both the tree and the screenshot come back regardless of\nwhat you pass (",{"type":43,"tag":56,"props":1346,"children":1348},{"className":1347},[],[1349],{"type":48,"value":1295},{"type":48,"value":278},{"type":43,"tag":56,"props":1352,"children":1354},{"className":1353},[],[1355],{"type":48,"value":1356},"vision",{"type":48,"value":278},{"type":43,"tag":56,"props":1359,"children":1361},{"className":1360},[],[1362],{"type":48,"value":1363},"som",{"type":48,"value":1365},", anything). There is no\n",{"type":43,"tag":56,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":48,"value":1295},{"type":48,"value":114},{"type":43,"tag":56,"props":1373,"children":1375},{"className":1374},[],[1376],{"type":48,"value":1356},{"type":48,"value":114},{"type":43,"tag":56,"props":1379,"children":1381},{"className":1380},[],[1382],{"type":48,"value":1363},{"type":48,"value":1384}," capture choice anymore. Drop the word \"vision\"\nfor perception entirely. (The tool named ",{"type":43,"tag":56,"props":1386,"children":1388},{"className":1387},[],[1389],{"type":48,"value":488},{"type":48,"value":1391}," is separate —\nraw PNG, no AX walk — and unrelated.)",{"type":43,"tag":416,"props":1393,"children":1395},{"id":1394},"the-modality-is-chosen-at-action-time-ax-vs-px",[1396,1398,1403,1405],{"type":48,"value":1397},"The modality is chosen at ACTION time — ",{"type":43,"tag":56,"props":1399,"children":1401},{"className":1400},[],[1402],{"type":48,"value":1295},{"type":48,"value":1404}," vs ",{"type":43,"tag":56,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":48,"value":1302},{"type":43,"tag":50,"props":1411,"children":1412},{},[1413,1415,1420],{"type":48,"value":1414},"You don't pick a capture mode; you pick ",{"type":43,"tag":77,"props":1416,"children":1417},{},[1418],{"type":48,"value":1419},"how you address the target",{"type":48,"value":1421},"\non the action call, and that one choice selects the rung:",{"type":43,"tag":85,"props":1423,"children":1424},{},[1425,1471],{"type":43,"tag":89,"props":1426,"children":1427},{},[1428,1433,1435,1440,1441,1447,1449,1454,1456,1462,1464,1469],{"type":43,"tag":77,"props":1429,"children":1430},{},[1431],{"type":48,"value":1432},"element ax action",{"type":48,"value":1434}," — pass ",{"type":43,"tag":56,"props":1436,"children":1438},{"className":1437},[],[1439],{"type":48,"value":619},{"type":48,"value":639},{"type":43,"tag":56,"props":1442,"children":1444},{"className":1443},[],[1445],{"type":48,"value":1446},"element_token",{"type":48,"value":1448},".\nDispatches through the ",{"type":43,"tag":77,"props":1450,"children":1451},{},[1452],{"type":48,"value":1453},"accessibility rung",{"type":48,"value":1455},": AXPress (macOS) \u002F UIA\nInvoke (Windows) \u002F AT-SPI ",{"type":43,"tag":56,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":48,"value":1461},"doAction",{"type":48,"value":1463}," (Linux). Backgroundable,\nz-order-independent, and the only ",{"type":43,"tag":77,"props":1465,"children":1466},{},[1467],{"type":48,"value":1468},"driver-verifiable",{"type":48,"value":1470}," rung.",{"type":43,"tag":89,"props":1472,"children":1473},{},[1474,1479,1480,1486,1487,1493,1495,1500,1502,1507],{"type":43,"tag":77,"props":1475,"children":1476},{},[1477],{"type":48,"value":1478},"element px action",{"type":48,"value":1434},{"type":43,"tag":56,"props":1481,"children":1483},{"className":1482},[],[1484],{"type":48,"value":1485},"x",{"type":48,"value":278},{"type":43,"tag":56,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":48,"value":1492},"y",{"type":48,"value":1494},". Dispatches through the ",{"type":43,"tag":77,"props":1496,"children":1497},{},[1498],{"type":48,"value":1499},"pixel\nrung",{"type":48,"value":1501},", reading the coordinate straight off the screenshot that's\nalready in the ",{"type":43,"tag":56,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":48,"value":1131},{"type":48,"value":1508}," response. Best-effort; the caller\nconfirms the effect.",{"type":43,"tag":50,"props":1510,"children":1511},{},[1512,1517,1519,1524,1525,1530,1532,1538,1540,1545,1547,1552,1553,1558],{"type":43,"tag":56,"props":1513,"children":1515},{"className":1514},[],[1516],{"type":48,"value":1295},{"type":48,"value":1518},"↔",{"type":43,"tag":56,"props":1520,"children":1522},{"className":1521},[],[1523],{"type":48,"value":619},{"type":48,"value":278},{"type":43,"tag":56,"props":1526,"children":1528},{"className":1527},[],[1529],{"type":48,"value":1302},{"type":48,"value":1531},"↔pixel ",{"type":43,"tag":56,"props":1533,"children":1535},{"className":1534},[],[1536],{"type":48,"value":1537},"x,y",{"type":48,"value":1539},". We retired the word \"vision\"\nfor the ",{"type":43,"tag":915,"props":1541,"children":1542},{},[1543],{"type":48,"value":1544},"dispatch",{"type":48,"value":1546}," path — it conflated perception with dispatch.\nPerception is always both; dispatch is ",{"type":43,"tag":56,"props":1548,"children":1550},{"className":1549},[],[1551],{"type":48,"value":1295},{"type":48,"value":521},{"type":43,"tag":56,"props":1554,"children":1556},{"className":1555},[],[1557],{"type":48,"value":1302},{"type":48,"value":209},{"type":43,"tag":50,"props":1560,"children":1561},{},[1562,1567,1568,1573,1574,1580,1582,1588,1590,1595,1597,1602,1603,1608,1610,1623,1625,1631,1633,1639,1641,1647,1649,1655],{"type":43,"tag":77,"props":1563,"children":1564},{},[1565],{"type":48,"value":1566},"The keyboard family has both forms too.",{"type":48,"value":1143},{"type":43,"tag":56,"props":1569,"children":1571},{"className":1570},[],[1572],{"type":48,"value":1182},{"type":48,"value":278},{"type":43,"tag":56,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":48,"value":1579},"press_key",{"type":48,"value":1581},",\nand ",{"type":43,"tag":56,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":48,"value":1587},"hotkey",{"type":48,"value":1589}," take ",{"type":43,"tag":56,"props":1591,"children":1593},{"className":1592},[],[1594],{"type":48,"value":619},{"type":48,"value":1596}," (ax) ",{"type":43,"tag":77,"props":1598,"children":1599},{},[1600],{"type":48,"value":1601},"or",{"type":48,"value":1143},{"type":43,"tag":56,"props":1604,"children":1606},{"className":1605},[],[1607],{"type":48,"value":1537},{"type":48,"value":1609}," (px) — mutually\nexclusive, same as the pointer tools. The px form ",{"type":43,"tag":77,"props":1611,"children":1612},{},[1613,1615,1621],{"type":48,"value":1614},"pixel-clicks at\n",{"type":43,"tag":56,"props":1616,"children":1618},{"className":1617},[],[1619],{"type":48,"value":1620},"(x,y)",{"type":48,"value":1622}," to establish real renderer focus, then delivers the\nkeystroke(s)",{"type":48,"value":1624}," to the now-focused element (it reuses ",{"type":43,"tag":56,"props":1626,"children":1628},{"className":1627},[],[1629],{"type":48,"value":1630},"click",{"type":48,"value":1632},"'s\ncoordinate translation + ",{"type":43,"tag":56,"props":1634,"children":1636},{"className":1635},[],[1637],{"type":48,"value":1638},"delivery_mode",{"type":48,"value":1640},"). That gives e.g.\n",{"type":43,"tag":56,"props":1642,"children":1644},{"className":1643},[],[1645],{"type":48,"value":1646},"type_text({pid, window_id, x, y, text})",{"type":48,"value":1648}," as a one-call focus-then-type\nfor Chromium\u002FElectron inputs the AX path can't reach, and\n",{"type":43,"tag":56,"props":1650,"children":1652},{"className":1651},[],[1653],{"type":48,"value":1654},"hotkey({pid, x, y, keys:[\"cmd\",\"v\"]})",{"type":48,"value":1656}," to paste into a specific field.",{"type":43,"tag":50,"props":1658,"children":1659},{},[1660,1665,1667,1672,1674,1679,1681,1687,1689,1695,1697,1703,1705,1710,1712,1717,1719,1724,1726,1732],{"type":43,"tag":77,"props":1661,"children":1662},{},[1663],{"type":48,"value":1664},"Typing default (the ladder).",{"type":48,"value":1666}," Call ",{"type":43,"tag":56,"props":1668,"children":1670},{"className":1669},[],[1671],{"type":48,"value":1182},{"type":48,"value":1673}," directly with\n",{"type":43,"tag":56,"props":1675,"children":1677},{"className":1676},[],[1678],{"type":48,"value":619},{"type":48,"value":1680}," (ax) — it targets the field, no pre-click. On\nElectron\u002FCatalyst the AX layer echoes the write without rendering it,\nso the driver returns ",{"type":43,"tag":56,"props":1682,"children":1684},{"className":1683},[],[1685],{"type":48,"value":1686},"effect:\"unverifiable\"",{"type":48,"value":1688}," + ",{"type":43,"tag":56,"props":1690,"children":1692},{"className":1691},[],[1693],{"type":48,"value":1694},"escalation:\"px\"",{"type":48,"value":1696},"\nthere (never a false ",{"type":43,"tag":56,"props":1698,"children":1700},{"className":1699},[],[1701],{"type":48,"value":1702},"verified:true",{"type":48,"value":1704},") — follow it, and cross-check the\nscreenshot in the response (the only ground truth). Escalate to the px\nform — ",{"type":43,"tag":56,"props":1706,"children":1708},{"className":1707},[],[1709],{"type":48,"value":1646},{"type":48,"value":1711}," — which pixel-clicks\nto focus, then types. ",{"type":43,"tag":77,"props":1713,"children":1714},{},[1715],{"type":48,"value":1716},"If the target control is closed",{"type":48,"value":1718}," (a search\nbutton, a collapsed field), AX-press to open it first (AX actions work\nin the background): a px focus-click won't reliably open ",{"type":43,"tag":915,"props":1720,"children":1721},{},[1722],{"type":48,"value":1723},"and",{"type":48,"value":1725}," focus a\nclosed control, so the text leaks into whatever's already focused.\nEscalate to ",{"type":43,"tag":56,"props":1727,"children":1729},{"className":1728},[],[1730],{"type":48,"value":1731},"delivery_mode:\"foreground\"",{"type":48,"value":1733}," only if it still drops.",{"type":43,"tag":50,"props":1735,"children":1736},{},[1737,1747,1749,1754,1756,1762,1764,1769,1770,1776,1778,1783,1785,1790,1792,1797,1798,1803],{"type":43,"tag":77,"props":1738,"children":1739},{},[1740,1745],{"type":43,"tag":56,"props":1741,"children":1743},{"className":1742},[],[1744],{"type":48,"value":188},{"type":48,"value":1746}," stays AX-only by design",{"type":48,"value":1748}," — it's for ",{"type":43,"tag":77,"props":1750,"children":1751},{},[1752],{"type":48,"value":1753},"non-text",{"type":48,"value":1755},"\ncontrols (dropdown \u002F ",{"type":43,"tag":56,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":48,"value":1761},"AXPopUpButton",{"type":48,"value":1763},", checkbox, slider, stepper). Its\npixel counterpart is a ",{"type":43,"tag":56,"props":1765,"children":1767},{"className":1766},[],[1768],{"type":48,"value":1630},{"type":48,"value":114},{"type":43,"tag":56,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":48,"value":1775},"drag",{"type":48,"value":1777}," on the control, not a \"set value\nat a pixel.\" So: text → ",{"type":43,"tag":56,"props":1779,"children":1781},{"className":1780},[],[1782],{"type":48,"value":1182},{"type":48,"value":1784}," (ax+px); non-text control values →\n",{"type":43,"tag":56,"props":1786,"children":1788},{"className":1787},[],[1789],{"type":48,"value":188},{"type":48,"value":1791},"; pixel-manipulate a control → ",{"type":43,"tag":56,"props":1793,"children":1795},{"className":1794},[],[1796],{"type":48,"value":1630},{"type":48,"value":114},{"type":43,"tag":56,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":48,"value":1775},{"type":48,"value":209},{"type":43,"tag":50,"props":1805,"children":1806},{},[1807],{"type":43,"tag":77,"props":1808,"children":1809},{},[1810],{"type":48,"value":1811},"Action responses carry an effect\u002Fescalation verdict",{"type":43,"tag":50,"props":1813,"children":1814},{},[1815,1817,1823],{"type":48,"value":1816},"Every action response keeps ",{"type":43,"tag":56,"props":1818,"children":1820},{"className":1819},[],[1821],{"type":48,"value":1822},"verified",{"type":48,"value":1824}," (did the driver read back a\npost-condition?) and adds two machine-readable fields so you know\nwhether — and where — to climb the ladder:",{"type":43,"tag":85,"props":1826,"children":1827},{},[1828,1924],{"type":43,"tag":89,"props":1829,"children":1830},{},[1831,1837,1839],{"type":43,"tag":56,"props":1832,"children":1834},{"className":1833},[],[1835],{"type":48,"value":1836},"effect",{"type":48,"value":1838},": one of\n",{"type":43,"tag":85,"props":1840,"children":1841},{},[1842,1860,1885],{"type":43,"tag":89,"props":1843,"children":1844},{},[1845,1851,1853,1858],{"type":43,"tag":56,"props":1846,"children":1848},{"className":1847},[],[1849],{"type":48,"value":1850},"\"confirmed\"",{"type":48,"value":1852}," — the driver read back the effect (",{"type":43,"tag":56,"props":1854,"children":1856},{"className":1855},[],[1857],{"type":48,"value":1295},{"type":48,"value":1859}," rung only).",{"type":43,"tag":89,"props":1861,"children":1862},{},[1863,1869,1871,1876,1878,1883],{"type":43,"tag":56,"props":1864,"children":1866},{"className":1865},[],[1867],{"type":48,"value":1868},"\"unverifiable\"",{"type":48,"value":1870}," — dispatched, but the driver has no handle to\nread back (every ",{"type":43,"tag":56,"props":1872,"children":1874},{"className":1873},[],[1875],{"type":48,"value":1302},{"type":48,"value":1877},"\u002FCGEvent path; foreground rung). ",{"type":43,"tag":77,"props":1879,"children":1880},{},[1881],{"type":48,"value":1882},"You",{"type":48,"value":1884},"\nconfirm it off the screenshot — it is not a failure.",{"type":43,"tag":89,"props":1886,"children":1887},{},[1888,1894,1896,1901,1903,1908,1910,1922],{"type":43,"tag":56,"props":1889,"children":1891},{"className":1890},[],[1892],{"type":48,"value":1893},"\"suspected_noop\"",{"type":48,"value":1895}," — the ",{"type":43,"tag":56,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":48,"value":1295},{"type":48,"value":1902}," action ",{"type":43,"tag":77,"props":1904,"children":1905},{},[1906],{"type":48,"value":1907},"likely did nothing",{"type":48,"value":1909}," (the\nelement didn't actually advertise the action, or you hit a passive\nlabel). This is the explicit ",{"type":43,"tag":77,"props":1911,"children":1912},{},[1913,1915,1920],{"type":48,"value":1914},"\"cross to ",{"type":43,"tag":56,"props":1916,"children":1918},{"className":1917},[],[1919],{"type":48,"value":1302},{"type":48,"value":1921},"\"",{"type":48,"value":1923}," trigger.",{"type":43,"tag":89,"props":1925,"children":1926},{},[1927,1933,1935,1941,1943],{"type":43,"tag":56,"props":1928,"children":1930},{"className":1929},[],[1931],{"type":48,"value":1932},"escalation",{"type":48,"value":1934},": ",{"type":43,"tag":56,"props":1936,"children":1938},{"className":1937},[],[1939],{"type":48,"value":1940},"{recommended, reason}",{"type":48,"value":1942}," when the driver thinks you\nshould change rung —\n",{"type":43,"tag":85,"props":1944,"children":1945},{},[1946,1970],{"type":43,"tag":89,"props":1947,"children":1948},{},[1949,1955,1957,1962,1964,1968],{"type":43,"tag":56,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":48,"value":1954},"\"px\"",{"type":48,"value":1956}," — the element isn't really actionable in ",{"type":43,"tag":56,"props":1958,"children":1960},{"className":1959},[],[1961],{"type":48,"value":1295},{"type":48,"value":1963},"; do an\n",{"type":43,"tag":77,"props":1965,"children":1966},{},[1967],{"type":48,"value":1478},{"type":48,"value":1969}," off the screenshot you already have.",{"type":43,"tag":89,"props":1971,"children":1972},{},[1973,1979,1981,1986,1988,1993],{"type":43,"tag":56,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":48,"value":1978},"\"foreground\"",{"type":48,"value":1980}," — a background insert\u002Fclick was ",{"type":43,"tag":915,"props":1982,"children":1983},{},[1984],{"type":48,"value":1985},"dropped",{"type":48,"value":1987}," on\ndelivery; re-call the same action with ",{"type":43,"tag":56,"props":1989,"children":1991},{"className":1990},[],[1992],{"type":48,"value":1731},{"type":48,"value":209},{"type":43,"tag":50,"props":1995,"children":1996},{},[1997,2002,2004,2010,2022,2024,2029],{"type":43,"tag":56,"props":1998,"children":2000},{"className":1999},[],[2001],{"type":48,"value":1131},{"type":48,"value":2003}," itself, when the AX tree comes back empty (a non-AX\nsurface like Electron\u002FChromium\u002Fcanvas), returns ",{"type":43,"tag":56,"props":2005,"children":2007},{"className":2006},[],[2008],{"type":48,"value":2009},"degraded: true",{"type":43,"tag":77,"props":2011,"children":2012},{},[2013,2015,2020],{"type":48,"value":2014},"plus the same ",{"type":43,"tag":56,"props":2016,"children":2018},{"className":2017},[],[2019],{"type":48,"value":1932},{"type":48,"value":2021}," hint",{"type":48,"value":2023}," — normally pointing at ",{"type":43,"tag":56,"props":2025,"children":2027},{"className":2026},[],[2028],{"type":48,"value":1302},{"type":48,"value":2030}," (you\nstill have the screenshot from the same call to click off).",{"type":43,"tag":50,"props":2032,"children":2033},{},[2034,2045,2047,2052,2054,2060,2062,2078,2080,2085,2087,2092,2094,2099,2100,2105],{"type":43,"tag":77,"props":2035,"children":2036},{},[2037,2039,2044],{"type":48,"value":2038},"Platform nuance for ",{"type":43,"tag":56,"props":2040,"children":2042},{"className":2041},[],[2043],{"type":48,"value":1932},{"type":48,"value":209},{"type":48,"value":2046}," On ",{"type":43,"tag":77,"props":2048,"children":2049},{},[2050],{"type":48,"value":2051},"Wayland",{"type":48,"value":2053}," an unfocused\nwindow cannot be pixel-targeted in the background (libei →\n",{"type":43,"tag":56,"props":2055,"children":2057},{"className":2056},[],[2058],{"type":48,"value":2059},"background_unavailable",{"type":48,"value":2061},"), so there the recommendation is\n",{"type":43,"tag":77,"props":2063,"children":2064},{},[2065,2071,2073],{"type":43,"tag":56,"props":2066,"children":2068},{"className":2067},[],[2069],{"type":48,"value":2070},"foreground",{"type":48,"value":2072},", not ",{"type":43,"tag":56,"props":2074,"children":2076},{"className":2075},[],[2077],{"type":48,"value":1302},{"type":48,"value":2079},". macOS, X11, and most Windows surfaces\n",{"type":43,"tag":915,"props":2081,"children":2082},{},[2083],{"type":48,"value":2084},"can",{"type":48,"value":2086}," pixel-target in the background, so they recommend ",{"type":43,"tag":56,"props":2088,"children":2090},{"className":2089},[],[2091],{"type":48,"value":1302},{"type":48,"value":2093},". See\n",{"type":43,"tag":56,"props":2095,"children":2097},{"className":2096},[],[2098],{"type":48,"value":161},{"type":48,"value":639},{"type":43,"tag":56,"props":2101,"children":2103},{"className":2102},[],[2104],{"type":48,"value":144},{"type":48,"value":209},{"type":43,"tag":65,"props":2107,"children":2109},{"id":2108},"the-verify-then-escalate-ladder-algorithm",[2110],{"type":48,"value":2111},"The verify-then-escalate ladder (algorithm)",{"type":43,"tag":50,"props":2113,"children":2114},{},[2115,2117,2122],{"type":48,"value":2116},"Every snapshot already hands you both the tree and the screenshot, so\nverifying never means \"go take a screenshot\" — it means cross-check\nthe tree against the pixels you already have, and only change\n",{"type":43,"tag":915,"props":2118,"children":2119},{},[2120],{"type":48,"value":2121},"dispatch rung",{"type":48,"value":2123}," on a real signal. Walk the rungs:",{"type":43,"tag":435,"props":2125,"children":2129},{"className":2126,"code":2128,"language":48},[2127],"language-text","# Rung 1 — element ax action, backgrounded (the cheap default)\nget_window_state(pid, window_id)            # tree + screenshot, both, always\nresp = click(pid, window_id, element_index) # or type_text \u002F set_value \u002F press_key\nget_window_state(pid, window_id)            # re-snapshot — did the tree change?\n\nif resp.effect == \"confirmed\" and tree changed:\n    done                                    # driver-verified\n\n# escalate only on a real signal\nif resp.effect == \"suspected_noop\"\n   or resp.escalation.recommended == \"px\"\n   or get_window_state.degraded            # empty tree → non-AX surface\n   or the tree looks wrong vs the screenshot:   # e.g. an h:1 \u002F off-viewport row\n\n    # Rung 2 — element px action off the SAME screenshot\n    pick the target pixel from the screenshot already in the response\n    click(pid, x, y)                        # background pixel — still no foreground\n    get_window_state(pid, window_id)        # re-snapshot, eyeball the result\n    if it landed: done\n\n# Rung 3 — background delivery was dropped (insert\u002Fclick never arrived)\nif resp.escalation.recommended == \"foreground\"\n   or the px action still did nothing:\n    re-call the same action with delivery_mode:\"foreground\"\n    # on Wayland this is the ONLY escalation — px-bg can't target an\n    # unfocused window there; see LINUX.md\n    verify again\n",[2130],{"type":43,"tag":56,"props":2131,"children":2132},{"__ignoreMap":440},[2133],{"type":48,"value":2128},{"type":43,"tag":50,"props":2135,"children":2136},{},[2137,2139,2143,2145,2151,2152,2158,2160,2164,2166,2171,2173,2178],{"type":48,"value":2138},"The two ideas to hold onto: (1) the AX tree ",{"type":43,"tag":77,"props":2140,"children":2141},{},[2142],{"type":48,"value":1155},{"type":48,"value":2144}," on canvas \u002F web \u002F\nCatalyst \u002F virtualized surfaces, so an unchanged-or-bogus tree plus\n",{"type":43,"tag":56,"props":2146,"children":2148},{"className":2147},[],[2149],{"type":48,"value":2150},"suspected_noop",{"type":48,"value":114},{"type":43,"tag":56,"props":2153,"children":2155},{"className":2154},[],[2156],{"type":48,"value":2157},"degraded",{"type":48,"value":2159}," — or a tree that simply disagrees with the\nscreenshot — is your cue to do an ",{"type":43,"tag":77,"props":2161,"children":2162},{},[2163],{"type":48,"value":1478},{"type":48,"value":2165}," off the\nscreenshot you already have; (2) ",{"type":43,"tag":56,"props":2167,"children":2169},{"className":2168},[],[2170],{"type":48,"value":1302},{"type":48,"value":2172}," is a ",{"type":43,"tag":915,"props":2174,"children":2175},{},[2176],{"type":48,"value":2177},"conscious",{"type":48,"value":2179}," switch to the\npixel addressing path, not a different capture.",{"type":43,"tag":50,"props":2181,"children":2182},{},[2183],{"type":43,"tag":77,"props":2184,"children":2185},{},[2186],{"type":48,"value":2187},"Window state → what works",{"type":43,"tag":2189,"props":2190,"children":2191},"table",{},[2192,2234],{"type":43,"tag":2193,"props":2194,"children":2195},"thead",{},[2196],{"type":43,"tag":2197,"props":2198,"children":2199},"tr",{},[2200,2206,2214,2219,2229],{"type":43,"tag":2201,"props":2202,"children":2203},"th",{},[2204],{"type":48,"value":2205},"state",{"type":43,"tag":2201,"props":2207,"children":2208},{},[2209],{"type":43,"tag":56,"props":2210,"children":2212},{"className":2211},[],[2213],{"type":48,"value":1131},{"type":43,"tag":2201,"props":2215,"children":2216},{},[2217],{"type":48,"value":2218},"element-index click (AX\u002FUIA)",{"type":43,"tag":2201,"props":2220,"children":2221},{},[2222,2227],{"type":43,"tag":56,"props":2223,"children":2225},{"className":2224},[],[2226],{"type":48,"value":1579},{"type":48,"value":2228}," commit",{"type":43,"tag":2201,"props":2230,"children":2231},{},[2232],{"type":48,"value":2233},"pixel click",{"type":43,"tag":2235,"props":2236,"children":2237},"tbody",{},[2238,2264,2288,2325,2351],{"type":43,"tag":2197,"props":2239,"children":2240},{},[2241,2247,2252,2256,2260],{"type":43,"tag":2242,"props":2243,"children":2244},"td",{},[2245],{"type":48,"value":2246},"frontmost",{"type":43,"tag":2242,"props":2248,"children":2249},{},[2250],{"type":48,"value":2251},"✅",{"type":43,"tag":2242,"props":2253,"children":2254},{},[2255],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2257,"children":2258},{},[2259],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2261,"children":2262},{},[2263],{"type":48,"value":2251},{"type":43,"tag":2197,"props":2265,"children":2266},{},[2267,2272,2276,2280,2284],{"type":43,"tag":2242,"props":2268,"children":2269},{},[2270],{"type":48,"value":2271},"backgrounded \u002F visible",{"type":43,"tag":2242,"props":2273,"children":2274},{},[2275],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2277,"children":2278},{},[2279],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2281,"children":2282},{},[2283],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2285,"children":2286},{},[2287],{"type":48,"value":2251},{"type":43,"tag":2197,"props":2289,"children":2290},{},[2291,2299,2303,2308,2320],{"type":43,"tag":2242,"props":2292,"children":2293},{},[2294],{"type":43,"tag":77,"props":2295,"children":2296},{},[2297],{"type":48,"value":2298},"minimized",{"type":43,"tag":2242,"props":2300,"children":2301},{},[2302],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2304,"children":2305},{},[2306],{"type":48,"value":2307},"✅ (actions fire in place)",{"type":43,"tag":2242,"props":2309,"children":2310},{},[2311,2313,2318],{"type":48,"value":2312},"❌ silent no-op — use ",{"type":43,"tag":56,"props":2314,"children":2316},{"className":2315},[],[2317],{"type":48,"value":188},{"type":48,"value":2319}," or click equivalent",{"type":43,"tag":2242,"props":2321,"children":2322},{},[2323],{"type":48,"value":2324},"❌ no on-screen bounds",{"type":43,"tag":2197,"props":2326,"children":2327},{},[2328,2333,2337,2341,2346],{"type":43,"tag":2242,"props":2329,"children":2330},{},[2331],{"type":48,"value":2332},"hidden",{"type":43,"tag":2242,"props":2334,"children":2335},{},[2336],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2338,"children":2339},{},[2340],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2342,"children":2343},{},[2344],{"type":48,"value":2345},"depends",{"type":43,"tag":2242,"props":2347,"children":2348},{},[2349],{"type":48,"value":2350},"❌",{"type":43,"tag":2197,"props":2352,"children":2353},{},[2354,2359,2372,2376,2380],{"type":43,"tag":2242,"props":2355,"children":2356},{},[2357],{"type":48,"value":2358},"on another desktop \u002F Space",{"type":43,"tag":2242,"props":2360,"children":2361},{},[2362,2364,2370],{"type":48,"value":2363},"⚠️ tree may be stripped on some apps — response carries ",{"type":43,"tag":56,"props":2365,"children":2367},{"className":2366},[],[2368],{"type":48,"value":2369},"off_space: true",{"type":48,"value":2371}," so you can detect it",{"type":43,"tag":2242,"props":2373,"children":2374},{},[2375],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2377,"children":2378},{},[2379],{"type":48,"value":2251},{"type":43,"tag":2242,"props":2381,"children":2382},{},[2383],{"type":48,"value":2384},"❌ not in current-desktop list",{"type":43,"tag":50,"props":2386,"children":2387},{},[2388,2393,2395,2400],{"type":43,"tag":77,"props":2389,"children":2390},{},[2391],{"type":48,"value":2392},"Critical cell — minimized + keyboard commit.",{"type":48,"value":2394}," The keystroke\nreaches the app but accessibility focus doesn't propagate to renderer\nfocus on a minimized window. Workarounds in order of preference:\n",{"type":43,"tag":56,"props":2396,"children":2398},{"className":2397},[],[2399],{"type":48,"value":188},{"type":48,"value":2401}," to write the field's entire value directly, or\nelement-index-click a commit-equivalent button (Go, Submit,\ncheckbox). Tell the user the window needs to un-minimize only as a\nlast resort.",{"type":43,"tag":65,"props":2403,"children":2405},{"id":2404},"the-canonical-loop",[2406],{"type":48,"value":2407},"The canonical loop",{"type":43,"tag":435,"props":2409,"children":2412},{"className":2410,"code":2411,"language":48},[2127],"start_session(session)            # once per run: declares this run's identity\nlaunch_app(target)\n  → pick window_id from the returned `windows` array\n    (or call list_windows(pid) separately)\n  → get_window_state(pid, window_id)\n    → [act]  # every action also takes (pid, window_id) + your `session`\n  → get_window_state(pid, window_id) → verify\nend_session(session)              # when the run finishes\n",[2413],{"type":43,"tag":56,"props":2414,"children":2415},{"__ignoreMap":440},[2416],{"type":48,"value":2411},{"type":43,"tag":50,"props":2418,"children":2419},{},[2420,2425,2427,2432,2434,2439,2441,2446,2448,2453],{"type":43,"tag":56,"props":2421,"children":2423},{"className":2422},[],[2424],{"type":48,"value":519},{"type":48,"value":2426}," now returns a ",{"type":43,"tag":56,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":48,"value":1104},{"type":48,"value":2433}," array alongside the pid, so the\ncommon case collapses to two calls (",{"type":43,"tag":56,"props":2435,"children":2437},{"className":2436},[],[2438],{"type":48,"value":519},{"type":48,"value":2440}," → ",{"type":43,"tag":56,"props":2442,"children":2444},{"className":2443},[],[2445],{"type":48,"value":1131},{"type":48,"value":2447},")\nwithout a separate ",{"type":43,"tag":56,"props":2449,"children":2451},{"className":2450},[],[2452],{"type":48,"value":527},{"type":48,"value":2454}," hop.",{"type":43,"tag":50,"props":2456,"children":2457},{},[2458,2463,2465,2470,2472,2478,2480,2486,2488,2494,2496,2501,2503,2509],{"type":43,"tag":77,"props":2459,"children":2460},{},[2461],{"type":48,"value":2462},"Declare a session.",{"type":48,"value":2464}," A session is ",{"type":43,"tag":915,"props":2466,"children":2467},{},[2468],{"type":48,"value":2469},"your run's",{"type":48,"value":2471}," identity — a stable id\nyou choose (",{"type":43,"tag":56,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":48,"value":2477},"\"research-1\"",{"type":48,"value":2479},"), declared with ",{"type":43,"tag":56,"props":2481,"children":2483},{"className":2482},[],[2484],{"type":48,"value":2485},"start_session",{"type":48,"value":2487}," and passed as\n",{"type":43,"tag":56,"props":2489,"children":2491},{"className":2490},[],[2492],{"type":48,"value":2493},"session",{"type":48,"value":2495}," on every action. It owns your agent cursor (a distinct colour\nper id), follows the run across any apps\u002Fwindows, and is the same whether\nyou drive over MCP, the CLI, or the socket. The cursor is ",{"type":43,"tag":77,"props":2497,"children":2498},{},[2499],{"type":48,"value":2500},"opt-in",{"type":48,"value":2502},": it\nappears only once you declare a session (anonymous actions run cursor-less).\nEnd with ",{"type":43,"tag":56,"props":2504,"children":2506},{"className":2505},[],[2507],{"type":48,"value":2508},"end_session",{"type":48,"value":2510}," (or the idle-TTL reclaims it).",{"type":43,"tag":50,"props":2512,"children":2513},{},[2514,2519,2520,2525,2527,2532,2534,2544,2546,2552,2554,2559,2561,2566,2568,2573],{"type":43,"tag":77,"props":2515,"children":2516},{},[2517],{"type":48,"value":2518},"Concurrent runs\u002Fsubagents:",{"type":48,"value":1143},{"type":43,"tag":56,"props":2521,"children":2523},{"className":2522},[],[2524],{"type":48,"value":519},{"type":48,"value":2526}," is idempotent — two runs that\nlaunch the same app get the ",{"type":43,"tag":77,"props":2528,"children":2529},{},[2530],{"type":48,"value":2531},"same",{"type":48,"value":2533}," instance (and on single-instance apps\nlike Calculator, the same window), so they clobber each other. Give each run\nits ",{"type":43,"tag":77,"props":2535,"children":2536},{},[2537,2539],{"type":48,"value":2538},"own ",{"type":43,"tag":56,"props":2540,"children":2542},{"className":2541},[],[2543],{"type":48,"value":2493},{"type":48,"value":2545}," (→ its own cursor) AND pass\n",{"type":43,"tag":56,"props":2547,"children":2549},{"className":2548},[],[2550],{"type":48,"value":2551},"creates_new_application_instance: true",{"type":48,"value":2553}," to ",{"type":43,"tag":56,"props":2555,"children":2557},{"className":2556},[],[2558],{"type":48,"value":519},{"type":48,"value":2560}," (→ its own window).\nThe element cache is keyed on ",{"type":43,"tag":56,"props":2562,"children":2564},{"className":2563},[],[2565],{"type":48,"value":1035},{"type":48,"value":2567}," and the cursor on ",{"type":43,"tag":56,"props":2569,"children":2571},{"className":2570},[],[2572],{"type":48,"value":2493},{"type":48,"value":2574},",\nso distinct instances + distinct sessions keep the runs fully separated.",{"type":43,"tag":50,"props":2576,"children":2577},{},[2578,2583,2585,2590,2592,2597,2599,2605,2607,2612,2614,2619,2621,2626,2628,2634,2635,2641,2643,2649],{"type":43,"tag":77,"props":2579,"children":2580},{},[2581],{"type":48,"value":2582},"Parallelism vs. ordering.",{"type":48,"value":2584}," Distinct sessions give distinct ",{"type":43,"tag":915,"props":2586,"children":2587},{},[2588],{"type":48,"value":2589},"cursors",{"type":48,"value":2591},", not\ndistinct ",{"type":43,"tag":915,"props":2593,"children":2594},{},[2595],{"type":48,"value":2596},"connections",{"type":48,"value":2598},". Subagents that share one ",{"type":43,"tag":56,"props":2600,"children":2602},{"className":2601},[],[2603],{"type":48,"value":2604},"cua-driver mcp",{"type":48,"value":2606}," (stdio)\nconnection have their tool calls ",{"type":43,"tag":77,"props":2608,"children":2609},{},[2610],{"type":48,"value":2611},"serialized",{"type":48,"value":2613}," by the transport — they take\nturns, not run in parallel. That's not a correctness problem (session + window\nisolation means they can't collide), just a throughput one. For genuinely\nparallel agents, give each its ",{"type":43,"tag":77,"props":2615,"children":2616},{},[2617],{"type":48,"value":2618},"own connection",{"type":48,"value":2620},": separate ",{"type":43,"tag":56,"props":2622,"children":2624},{"className":2623},[],[2625],{"type":48,"value":2604},{"type":48,"value":2627},"\nprocesses, or point each agent's MCP client at the daemon's HTTP endpoint\n(",{"type":43,"tag":56,"props":2629,"children":2631},{"className":2630},[],[2632],{"type":48,"value":2633},"CUA_DRIVER_RS_MCP_HTTP_PORT",{"type":48,"value":2440},{"type":43,"tag":56,"props":2636,"children":2638},{"className":2637},[],[2639],{"type":48,"value":2640},"POST http:\u002F\u002F127.0.0.1:\u003Cport>\u002Fmcp",{"type":48,"value":2642},"). The daemon\nserves connections concurrently; per-connection ordering keeps each agent's own\nsequence (e.g. ",{"type":43,"tag":56,"props":2644,"children":2646},{"className":2645},[],[2647],{"type":48,"value":2648},"3 → + → 1 → =",{"type":48,"value":2650},") correct.",{"type":43,"tag":50,"props":2652,"children":2653},{},[2654,2660,2662,2667,2669,2674,2676,2681,2683,2688,2690,2695],{"type":43,"tag":56,"props":2655,"children":2657},{"className":2656},[],[2658],{"type":48,"value":2659},"list_apps",{"type":48,"value":2661}," is for app-level discovery (answering \"what's installed \u002F\nrunning \u002F frontmost?\") — not part of the core action loop. Skip it\nin the loop. For ",{"type":43,"tag":77,"props":2663,"children":2664},{},[2665],{"type":48,"value":2666},"window-level",{"type":48,"value":2668}," questions — \"does this app have a\nvisible window?\", \"which desktop is this window on?\", \"which of this\npid's windows is the main one?\" — call ",{"type":43,"tag":56,"props":2670,"children":2672},{"className":2671},[],[2673],{"type":48,"value":527},{"type":48,"value":2675}," instead; the\napp record doesn't carry window state on purpose. In the common\nsingle-window case you can skip ",{"type":43,"tag":56,"props":2677,"children":2679},{"className":2678},[],[2680],{"type":48,"value":527},{"type":48,"value":2682}," entirely and read the\n",{"type":43,"tag":56,"props":2684,"children":2686},{"className":2685},[],[2687],{"type":48,"value":1104},{"type":48,"value":2689}," array that ",{"type":43,"tag":56,"props":2691,"children":2693},{"className":2692},[],[2694],{"type":48,"value":519},{"type":48,"value":2696}," already returned.",{"type":43,"tag":416,"props":2698,"children":2700},{"id":2699},"snapshot-and-act-by-element_index",[2701],{"type":48,"value":2702},"Snapshot and act by element_index",{"type":43,"tag":50,"props":2704,"children":2705},{},[2706,2708,2714,2716,2721,2723,2728,2730,2735,2737,2743,2745,2750,2752,2757,2759,2764],{"type":48,"value":2707},"Call ",{"type":43,"tag":56,"props":2709,"children":2711},{"className":2710},[],[2712],{"type":48,"value":2713},"get_window_state({pid, window_id})",{"type":48,"value":2715}," with the ",{"type":43,"tag":56,"props":2717,"children":2719},{"className":2718},[],[2720],{"type":48,"value":511},{"type":48,"value":2722}," from\n",{"type":43,"tag":56,"props":2724,"children":2726},{"className":2725},[],[2727],{"type":48,"value":519},{"type":48,"value":2729},"'s ",{"type":43,"tag":56,"props":2731,"children":2733},{"className":2732},[],[2734],{"type":48,"value":1104},{"type":48,"value":2736}," array (or a fresh ",{"type":43,"tag":56,"props":2738,"children":2740},{"className":2739},[],[2741],{"type":48,"value":2742},"list_windows({pid})",{"type":48,"value":2744}," if\nyou're interacting with a long-lived process). It returns ",{"type":43,"tag":77,"props":2746,"children":2747},{},[2748],{"type":48,"value":2749},"the tree\nand the screenshot together",{"type":48,"value":2751}," by default, so you can both dispatch by\n",{"type":43,"tag":56,"props":2753,"children":2755},{"className":2754},[],[2756],{"type":48,"value":619},{"type":48,"value":2758}," and ground on pixels from one call — no config change,\nno mode flip. When you're just re-indexing before an element ax action\nand don't need fresh pixels, pass ",{"type":43,"tag":56,"props":2760,"children":2762},{"className":2761},[],[2763],{"type":48,"value":1280},{"type":48,"value":2765}," to skip\nthe grab (a perf knob, not a modality choice).",{"type":43,"tag":50,"props":2767,"children":2768},{},[2769],{"type":48,"value":2770},"The response carries:",{"type":43,"tag":85,"props":2772,"children":2773},{},[2774,2830,2869,2888],{"type":43,"tag":89,"props":2775,"children":2776},{},[2777,2783,2785,2791,2793,2799,2801,2806,2808,2813,2815,2821,2822,2828],{"type":43,"tag":56,"props":2778,"children":2780},{"className":2779},[],[2781],{"type":48,"value":2782},"tree_markdown",{"type":48,"value":2784}," — every actionable element tagged ",{"type":43,"tag":56,"props":2786,"children":2788},{"className":2787},[],[2789],{"type":48,"value":2790},"[N]",{"type":48,"value":2792},". That ",{"type":43,"tag":56,"props":2794,"children":2796},{"className":2795},[],[2797],{"type":48,"value":2798},"N",{"type":48,"value":2800},"\nis the ",{"type":43,"tag":56,"props":2802,"children":2804},{"className":2803},[],[2805],{"type":48,"value":619},{"type":48,"value":2807},". The tree can be very large (Finder is\n~1600 elements, ~190 KB); when it exceeds token limits the MCP\nharness saves it to a file and returns the path. Use ",{"type":43,"tag":56,"props":2809,"children":2811},{"className":2810},[],[2812],{"type":48,"value":368},{"type":48,"value":2814}," +\n",{"type":43,"tag":56,"props":2816,"children":2818},{"className":2817},[],[2819],{"type":48,"value":2820},"jq -r '.tree_markdown'",{"type":48,"value":1688},{"type":43,"tag":56,"props":2823,"children":2825},{"className":2824},[],[2826],{"type":48,"value":2827},"grep",{"type":48,"value":2829}," to pull the section you need.",{"type":43,"tag":89,"props":2831,"children":2832},{},[2833,2838,2839,2844,2845,2850,2852,2857,2859,2867],{"type":43,"tag":56,"props":2834,"children":2836},{"className":2835},[],[2837],{"type":48,"value":1836},{"type":48,"value":639},{"type":43,"tag":56,"props":2840,"children":2842},{"className":2841},[],[2843],{"type":48,"value":1932},{"type":48,"value":639},{"type":43,"tag":56,"props":2846,"children":2848},{"className":2847},[],[2849],{"type":48,"value":2157},{"type":48,"value":2851}," — the verify-then-escalate\nsignals (see the behavior matrix above): ",{"type":43,"tag":56,"props":2853,"children":2855},{"className":2854},[],[2856],{"type":48,"value":2009},{"type":48,"value":2858}," means the\ntree came back empty (non-AX surface), so you act by ",{"type":43,"tag":77,"props":2860,"children":2861},{},[2862],{"type":43,"tag":56,"props":2863,"children":2865},{"className":2864},[],[2866],{"type":48,"value":1302},{"type":48,"value":2868}," off the\nscreenshot in the same response.",{"type":43,"tag":89,"props":2870,"children":2871},{},[2872,2878,2880,2886],{"type":43,"tag":56,"props":2873,"children":2875},{"className":2874},[],[2876],{"type":48,"value":2877},"screenshot_file_path",{"type":48,"value":2879}," — present when the screenshot was written to\ndisk instead of inlined (you passed ",{"type":43,"tag":56,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":48,"value":2885},"screenshot_out_file",{"type":48,"value":2887},", or the\ncontext-saving CLI path); otherwise the frame is inlined.",{"type":43,"tag":89,"props":2889,"children":2890},{},[2891,2897,2898,2904,2905,2911,2913,2918],{"type":43,"tag":56,"props":2892,"children":2894},{"className":2893},[],[2895],{"type":48,"value":2896},"screenshot_width",{"type":48,"value":639},{"type":43,"tag":56,"props":2899,"children":2901},{"className":2900},[],[2902],{"type":48,"value":2903},"_height",{"type":48,"value":639},{"type":43,"tag":56,"props":2906,"children":2908},{"className":2907},[],[2909],{"type":48,"value":2910},"_scale_factor",{"type":48,"value":2912}," — dimensions of\nthe captured image. Present whenever a screenshot was taken (i.e.\nunless you passed ",{"type":43,"tag":56,"props":2914,"children":2916},{"className":2915},[],[2917],{"type":48,"value":1280},{"type":48,"value":2919},").",{"type":43,"tag":50,"props":2921,"children":2922},{},[2923],{"type":43,"tag":77,"props":2924,"children":2925},{},[2926],{"type":48,"value":2927},"Getting the screenshot as a file (CLI and context-constrained agents):",{"type":43,"tag":435,"props":2929,"children":2931},{"className":437,"code":2930,"language":439,"meta":440,"style":440},"# write to file — stdout stays readable (AX\u002FUIA tree \u002F summary only, no base64)\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W,\"screenshot_out_file\":\"\u002Ftmp\u002Fshot.jpg\"}'\n\n# CLI --screenshot-out-file flag is equivalent\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W}' --screenshot-out-file \u002Ftmp\u002Fshot.jpg\n",[2932],{"type":43,"tag":56,"props":2933,"children":2934},{"__ignoreMap":440},[2935,2943,2967,2976,2984],{"type":43,"tag":446,"props":2936,"children":2937},{"class":448,"line":449},[2938],{"type":43,"tag":446,"props":2939,"children":2940},{"style":474},[2941],{"type":48,"value":2942},"# write to file — stdout stays readable (AX\u002FUIA tree \u002F summary only, no base64)\n",{"type":43,"tag":446,"props":2944,"children":2945},{"class":448,"line":731},[2946,2950,2954,2958,2963],{"type":43,"tag":446,"props":2947,"children":2948},{"style":453},[2949],{"type":48,"value":61},{"type":43,"tag":446,"props":2951,"children":2952},{"style":458},[2953],{"type":48,"value":779},{"type":43,"tag":446,"props":2955,"children":2956},{"style":744},[2957],{"type":48,"value":747},{"type":43,"tag":446,"props":2959,"children":2960},{"style":458},[2961],{"type":48,"value":2962},"{\"pid\":N,\"window_id\":W,\"screenshot_out_file\":\"\u002Ftmp\u002Fshot.jpg\"}",{"type":43,"tag":446,"props":2964,"children":2965},{"style":744},[2966],{"type":48,"value":757},{"type":43,"tag":446,"props":2968,"children":2969},{"class":448,"line":760},[2970],{"type":43,"tag":446,"props":2971,"children":2973},{"emptyLinePlaceholder":2972},true,[2974],{"type":48,"value":2975},"\n",{"type":43,"tag":446,"props":2977,"children":2978},{"class":448,"line":769},[2979],{"type":43,"tag":446,"props":2980,"children":2981},{"style":474},[2982],{"type":48,"value":2983},"# CLI --screenshot-out-file flag is equivalent\n",{"type":43,"tag":446,"props":2985,"children":2986},{"class":448,"line":795},[2987,2991,2995,2999,3004,3009,3014],{"type":43,"tag":446,"props":2988,"children":2989},{"style":453},[2990],{"type":48,"value":61},{"type":43,"tag":446,"props":2992,"children":2993},{"style":458},[2994],{"type":48,"value":779},{"type":43,"tag":446,"props":2996,"children":2997},{"style":744},[2998],{"type":48,"value":747},{"type":43,"tag":446,"props":3000,"children":3001},{"style":458},[3002],{"type":48,"value":3003},"{\"pid\":N,\"window_id\":W}",{"type":43,"tag":446,"props":3005,"children":3006},{"style":744},[3007],{"type":48,"value":3008},"'",{"type":43,"tag":446,"props":3010,"children":3011},{"style":458},[3012],{"type":48,"value":3013}," --screenshot-out-file",{"type":43,"tag":446,"props":3015,"children":3016},{"style":458},[3017],{"type":48,"value":3018}," \u002Ftmp\u002Fshot.jpg\n",{"type":43,"tag":50,"props":3020,"children":3021},{},[3022,3024,3029,3031,3036,3038,3043],{"type":48,"value":3023},"Pass ",{"type":43,"tag":56,"props":3025,"children":3027},{"className":3026},[],[3028],{"type":48,"value":2885},{"type":48,"value":3030}," when using ",{"type":43,"tag":56,"props":3032,"children":3034},{"className":3033},[],[3035],{"type":48,"value":1131},{"type":48,"value":3037}," via CLI or\nfrom an agent whose context window can't absorb ~31 KB of inline\nbase64 (e.g. OpenCode with a local Ollama model). The MCP image\ncontent block is omitted from the response when this param is set —\nthe model receives only the tree and ",{"type":43,"tag":56,"props":3039,"children":3041},{"className":3040},[],[3042],{"type":48,"value":2877},{"type":48,"value":3044},", then\nreads the image from disk.",{"type":43,"tag":50,"props":3046,"children":3047},{},[3048,3059],{"type":43,"tag":77,"props":3049,"children":3050},{},[3051,3053,3057],{"type":48,"value":3052},"The tree and the screenshot are complementary, not redundant — and\nthey come from the ",{"type":43,"tag":915,"props":3054,"children":3055},{},[3056],{"type":48,"value":2531},{"type":48,"value":3058}," call.",{"type":48,"value":3060}," Each half carries signal the other\ncan't, which is exactly why you cross-check them:",{"type":43,"tag":85,"props":3062,"children":3063},{},[3064,3095],{"type":43,"tag":89,"props":3065,"children":3066},{},[3067,3069,3074,3076,3081,3083,3088,3090,3094],{"type":48,"value":3068},"The ",{"type":43,"tag":77,"props":3070,"children":3071},{},[3072],{"type":48,"value":3073},"tree",{"type":48,"value":3075}," tells you ",{"type":43,"tag":915,"props":3077,"children":3078},{},[3079],{"type":48,"value":3080},"what's clickable",{"type":48,"value":3082}," — roles, labels,\n",{"type":43,"tag":56,"props":3084,"children":3086},{"className":3085},[],[3087],{"type":48,"value":619},{"type":48,"value":3089}," handles, advertised actions, parent-child\nstructure. This is the ground truth for an ",{"type":43,"tag":77,"props":3091,"children":3092},{},[3093],{"type":48,"value":1432},{"type":48,"value":209},{"type":43,"tag":89,"props":3096,"children":3097},{},[3098,3099,3103,3104,3109,3111,3116,3118,3123],{"type":48,"value":3068},{"type":43,"tag":77,"props":3100,"children":3101},{},[3102],{"type":48,"value":488},{"type":48,"value":3075},{"type":43,"tag":915,"props":3105,"children":3106},{},[3107],{"type":48,"value":3108},"which one",{"type":48,"value":3110}," — the tree often has many\nbuttons with similar or empty labels (\"Delete\", \"OK\", anonymous\nUUID-labeled buttons, repeated static-text), and visual context\ndisambiguates. Captions, colors, layout relationships visible in\npixels often don't show up in the tree at all (especially in\nChromium \u002F Electron \u002F web content) — and the screenshot is where you\ncatch the tree ",{"type":43,"tag":915,"props":3112,"children":3113},{},[3114],{"type":48,"value":3115},"lying",{"type":48,"value":3117}," (an ",{"type":43,"tag":56,"props":3119,"children":3121},{"className":3120},[],[3122],{"type":48,"value":1218},{"type":48,"value":3124},"\u002Foff-viewport row, a Catalyst null\nvalue).",{"type":43,"tag":50,"props":3126,"children":3127},{},[3128,3130,3135,3137,3141,3143,3147,3149,3154,3156,3161,3163,3168,3170,3175],{"type":48,"value":3129},"Default to dispatching by ",{"type":43,"tag":56,"props":3131,"children":3133},{"className":3132},[],[3134],{"type":48,"value":619},{"type":48,"value":3136}," (the ",{"type":43,"tag":77,"props":3138,"children":3139},{},[3140],{"type":48,"value":1432},{"type":48,"value":3142},") —\nit's the verifiable, backgroundable rung. Do an ",{"type":43,"tag":77,"props":3144,"children":3145},{},[3146],{"type":48,"value":1478},{"type":48,"value":3148},"\n(",{"type":43,"tag":56,"props":3150,"children":3152},{"className":3151},[],[3153],{"type":48,"value":1537},{"type":48,"value":3155}," off the same screenshot) when the tree can't disambiguate\n(repeated\u002Fempty labels), when it's empty (",{"type":43,"tag":56,"props":3157,"children":3159},{"className":3158},[],[3160],{"type":48,"value":2157},{"type":48,"value":3162}," — non-AX\nsurface), when an action came back ",{"type":43,"tag":56,"props":3164,"children":3166},{"className":3165},[],[3167],{"type":48,"value":2150},{"type":48,"value":3169},", or when the tree\ndisagrees with the pixels. You never re-capture to switch — the\nscreenshot is already there; you just change ",{"type":43,"tag":915,"props":3171,"children":3172},{},[3173],{"type":48,"value":3174},"how you address",{"type":48,"value":3176}," the\ntarget.",{"type":43,"tag":50,"props":3178,"children":3179},{},[3180],{"type":48,"value":3181},"Reach for pixel coordinates only when the target is a canvas \u002F\nvideo \u002F WebGL \u002F custom-drawn surface that isn't in the tree\n(see \"Pixel-coordinate clicks\" below).",{"type":43,"tag":50,"props":3183,"children":3184},{},[3185,3186,3192,3194,3199,3201,3207,3209,3214],{"type":48,"value":3068},{"type":43,"tag":56,"props":3187,"children":3189},{"className":3188},[],[3190],{"type":48,"value":3191},"actions=[...]",{"type":48,"value":3193}," list on each element is ",{"type":43,"tag":77,"props":3195,"children":3196},{},[3197],{"type":48,"value":3198},"advisory",{"type":48,"value":3200},", not\nauthoritative. cua-driver does not pre-flight check against it —\n",{"type":43,"tag":56,"props":3202,"children":3204},{"className":3203},[],[3205],{"type":48,"value":3206},"click({pid, element_index})",{"type":48,"value":3208}," always attempts the default action (or\nthe action you pass) and surfaces whatever the target returns. ",{"type":43,"tag":77,"props":3210,"children":3211},{},[3212],{"type":48,"value":3213},"Try\nthe click first",{"type":48,"value":3215}," — pivot only on the returned error code.",{"type":43,"tag":416,"props":3217,"children":3219},{"id":3218},"tool-dispatch-table",[3220],{"type":48,"value":3221},"Tool dispatch table",{"type":43,"tag":50,"props":3223,"children":3224},{},[3225,3227,3232,3234,3239,3241,3246,3248,3253],{"type":48,"value":3226},"Every row assumes a ",{"type":43,"tag":56,"props":3228,"children":3230},{"className":3229},[],[3231],{"type":48,"value":1035},{"type":48,"value":3233}," pair from the last\n",{"type":43,"tag":56,"props":3235,"children":3237},{"className":3236},[],[3238],{"type":48,"value":1131},{"type":48,"value":3240},"; ",{"type":43,"tag":56,"props":3242,"children":3244},{"className":3243},[],[3245],{"type":48,"value":511},{"type":48,"value":3247}," is required alongside ",{"type":43,"tag":56,"props":3249,"children":3251},{"className":3250},[],[3252],{"type":48,"value":619},{"type":48,"value":3254},",\nignored on pixel-only forms unless you want to anchor the conversion\nagainst a specific window.",{"type":43,"tag":2189,"props":3256,"children":3257},{},[3258,3279],{"type":43,"tag":2193,"props":3259,"children":3260},{},[3261],{"type":43,"tag":2197,"props":3262,"children":3263},{},[3264,3269,3274],{"type":43,"tag":2201,"props":3265,"children":3266},{},[3267],{"type":48,"value":3268},"Intent",{"type":43,"tag":2201,"props":3270,"children":3271},{},[3272],{"type":48,"value":3273},"Tool",{"type":43,"tag":2201,"props":3275,"children":3276},{},[3277],{"type":48,"value":3278},"Notes",{"type":43,"tag":2235,"props":3280,"children":3281},{},[3282,3352,3393,3437,3459,3493,3538,3597,3619,3668,3690],{"type":43,"tag":2197,"props":3283,"children":3284},{},[3285,3290,3298],{"type":43,"tag":2242,"props":3286,"children":3287},{},[3288],{"type":48,"value":3289},"List an app's windows",{"type":43,"tag":2242,"props":3291,"children":3292},{},[3293],{"type":43,"tag":56,"props":3294,"children":3296},{"className":3295},[],[3297],{"type":48,"value":2742},{"type":43,"tag":2242,"props":3299,"children":3300},{},[3301,3303,3308,3309,3315,3316,3322,3323,3329,3330,3336,3337,3343,3345,3350],{"type":48,"value":3302},"returns ",{"type":43,"tag":56,"props":3304,"children":3306},{"className":3305},[],[3307],{"type":48,"value":511},{"type":48,"value":278},{"type":43,"tag":56,"props":3310,"children":3312},{"className":3311},[],[3313],{"type":48,"value":3314},"title",{"type":48,"value":278},{"type":43,"tag":56,"props":3317,"children":3319},{"className":3318},[],[3320],{"type":48,"value":3321},"bounds",{"type":48,"value":278},{"type":43,"tag":56,"props":3324,"children":3326},{"className":3325},[],[3327],{"type":48,"value":3328},"z_index",{"type":48,"value":278},{"type":43,"tag":56,"props":3331,"children":3333},{"className":3332},[],[3334],{"type":48,"value":3335},"is_on_screen",{"type":48,"value":278},{"type":43,"tag":56,"props":3338,"children":3340},{"className":3339},[],[3341],{"type":48,"value":3342},"on_current_space",{"type":48,"value":3344},". Already included in ",{"type":43,"tag":56,"props":3346,"children":3348},{"className":3347},[],[3349],{"type":48,"value":519},{"type":48,"value":3351},"'s response — only call this for long-lived pids",{"type":43,"tag":2197,"props":3353,"children":3354},{},[3355,3360,3368],{"type":43,"tag":2242,"props":3356,"children":3357},{},[3358],{"type":48,"value":3359},"Snapshot a window",{"type":43,"tag":2242,"props":3361,"children":3362},{},[3363],{"type":43,"tag":56,"props":3364,"children":3366},{"className":3365},[],[3367],{"type":48,"value":2713},{"type":43,"tag":2242,"props":3369,"children":3370},{},[3371,3372,3377,3378,3384,3386,3391],{"type":48,"value":3302},{"type":43,"tag":56,"props":3373,"children":3375},{"className":3374},[],[3376],{"type":48,"value":2782},{"type":48,"value":1688},{"type":43,"tag":56,"props":3379,"children":3381},{"className":3380},[],[3382],{"type":48,"value":3383},"screenshot_*",{"type":48,"value":3385},"; populates the ",{"type":43,"tag":56,"props":3387,"children":3389},{"className":3388},[],[3390],{"type":48,"value":1035},{"type":48,"value":3392}," element_index cache",{"type":43,"tag":2197,"props":3394,"children":3395},{},[3396,3401,3410],{"type":43,"tag":2242,"props":3397,"children":3398},{},[3399],{"type":48,"value":3400},"Left click",{"type":43,"tag":2242,"props":3402,"children":3403},{},[3404],{"type":43,"tag":56,"props":3405,"children":3407},{"className":3406},[],[3408],{"type":48,"value":3409},"click({pid, window_id, element_index})",{"type":43,"tag":2242,"props":3411,"children":3412},{},[3413,3415,3421,3423,3429,3431],{"type":48,"value":3414},"default ",{"type":43,"tag":56,"props":3416,"children":3418},{"className":3417},[],[3419],{"type":48,"value":3420},"action: \"press\"",{"type":48,"value":3422},". Pixel form: ",{"type":43,"tag":56,"props":3424,"children":3426},{"className":3425},[],[3427],{"type":48,"value":3428},"click({pid, x, y})",{"type":48,"value":3430}," (window_id optional) — ",{"type":43,"tag":56,"props":3432,"children":3434},{"className":3433},[],[3435],{"type":48,"value":3436},"modifier: [\"cmd\"|\"ctrl\"]",{"type":43,"tag":2197,"props":3438,"children":3439},{},[3440,3445,3454],{"type":43,"tag":2242,"props":3441,"children":3442},{},[3443],{"type":48,"value":3444},"Double-click \u002F open",{"type":43,"tag":2242,"props":3446,"children":3447},{},[3448],{"type":43,"tag":56,"props":3449,"children":3451},{"className":3450},[],[3452],{"type":48,"value":3453},"double_click({pid, window_id, element_index})",{"type":43,"tag":2242,"props":3455,"children":3456},{},[3457],{"type":48,"value":3458},"Default action when the element advertises one (Open on Finder items \u002F openable rows), else stamped pixel double-click at the element's center",{"type":43,"tag":2197,"props":3460,"children":3461},{},[3462,3467,3483],{"type":43,"tag":2242,"props":3463,"children":3464},{},[3465],{"type":48,"value":3466},"Right click \u002F context menu",{"type":43,"tag":2242,"props":3468,"children":3469},{},[3470,3476,3477],{"type":43,"tag":56,"props":3471,"children":3473},{"className":3472},[],[3474],{"type":48,"value":3475},"right_click({pid, window_id, element_index})",{"type":48,"value":521},{"type":43,"tag":56,"props":3478,"children":3480},{"className":3479},[],[3481],{"type":48,"value":3482},"click({pid, window_id, element_index, action: \"show_menu\"})",{"type":43,"tag":2242,"props":3484,"children":3485},{},[3486,3488],{"type":48,"value":3487},"Chromium web-content coerces pixel right-click to left on macOS — see ",{"type":43,"tag":56,"props":3489,"children":3491},{"className":3490},[],[3492],{"type":48,"value":180},{"type":43,"tag":2197,"props":3494,"children":3495},{},[3496,3501,3520],{"type":43,"tag":2242,"props":3497,"children":3498},{},[3499],{"type":48,"value":3500},"Type at cursor",{"type":43,"tag":2242,"props":3502,"children":3503},{},[3504,3510,3512,3518],{"type":43,"tag":56,"props":3505,"children":3507},{"className":3506},[],[3508],{"type":48,"value":3509},"type_text({pid, text, window_id, element_index})",{"type":48,"value":3511}," (ax) or ",{"type":43,"tag":56,"props":3513,"children":3515},{"className":3514},[],[3516],{"type":48,"value":3517},"type_text({pid, text, window_id, x, y})",{"type":48,"value":3519}," (px)",{"type":43,"tag":2242,"props":3521,"children":3522},{},[3523,3525,3529,3531,3536],{"type":48,"value":3524},"ax focuses the element then writes via the platform's text-set primitive; ",{"type":43,"tag":77,"props":3526,"children":3527},{},[3528],{"type":48,"value":1302},{"type":48,"value":3530}," pixel-clicks ",{"type":43,"tag":56,"props":3532,"children":3534},{"className":3533},[],[3535],{"type":48,"value":1620},{"type":48,"value":3537}," to focus the renderer, then types — the one-call fix for Chromium\u002FElectron inputs the AX path can't reach",{"type":43,"tag":2197,"props":3539,"children":3540},{},[3541,3546,3555],{"type":43,"tag":2242,"props":3542,"children":3543},{},[3544],{"type":48,"value":3545},"Set whole non-text control value",{"type":43,"tag":2242,"props":3547,"children":3548},{},[3549],{"type":43,"tag":56,"props":3550,"children":3552},{"className":3551},[],[3553],{"type":48,"value":3554},"set_value({pid, window_id, element_index, value})",{"type":43,"tag":2242,"props":3556,"children":3557},{},[3558,3563,3565,3570,3572,3577,3579,3584,3586,3591,3592],{"type":43,"tag":77,"props":3559,"children":3560},{},[3561],{"type":48,"value":3562},"AX-only by design",{"type":48,"value":3564}," — dropdown\u002F",{"type":43,"tag":56,"props":3566,"children":3568},{"className":3567},[],[3569],{"type":48,"value":1761},{"type":48,"value":3571},", checkbox, slider, stepper; ",{"type":43,"tag":77,"props":3573,"children":3574},{},[3575],{"type":48,"value":3576},"also the keyboard-commit workaround on minimized windows.",{"type":48,"value":3578}," For text use ",{"type":43,"tag":56,"props":3580,"children":3582},{"className":3581},[],[3583],{"type":48,"value":1182},{"type":48,"value":3585},"; to pixel-manipulate a control use ",{"type":43,"tag":56,"props":3587,"children":3589},{"className":3588},[],[3590],{"type":48,"value":1630},{"type":48,"value":114},{"type":43,"tag":56,"props":3593,"children":3595},{"className":3594},[],[3596],{"type":48,"value":1775},{"type":43,"tag":2197,"props":3598,"children":3599},{},[3600,3605,3614],{"type":43,"tag":2242,"props":3601,"children":3602},{},[3603],{"type":48,"value":3604},"Scroll",{"type":43,"tag":2242,"props":3606,"children":3607},{},[3608],{"type":43,"tag":56,"props":3609,"children":3611},{"className":3610},[],[3612],{"type":48,"value":3613},"scroll({pid, direction, amount, by, window_id, element_index})",{"type":43,"tag":2242,"props":3615,"children":3616},{},[3617],{"type":48,"value":3618},"synthesizes per-pid PageUp\u002FPageDown\u002Farrows",{"type":43,"tag":2197,"props":3620,"children":3621},{},[3622,3627,3644],{"type":43,"tag":2242,"props":3623,"children":3624},{},[3625],{"type":48,"value":3626},"Focus + send key",{"type":43,"tag":2242,"props":3628,"children":3629},{},[3630,3636,3637,3643],{"type":43,"tag":56,"props":3631,"children":3633},{"className":3632},[],[3634],{"type":48,"value":3635},"press_key({pid, key, window_id, element_index, modifiers})",{"type":48,"value":3511},{"type":43,"tag":56,"props":3638,"children":3640},{"className":3639},[],[3641],{"type":48,"value":3642},"press_key({pid, key, x, y})",{"type":48,"value":3519},{"type":43,"tag":2242,"props":3645,"children":3646},{},[3647,3649,3654,3656,3660,3661,3666],{"type":48,"value":3648},"ax ",{"type":43,"tag":56,"props":3650,"children":3652},{"className":3651},[],[3653],{"type":48,"value":619},{"type":48,"value":3655}," sets focus then posts the key; ",{"type":43,"tag":77,"props":3657,"children":3658},{},[3659],{"type":48,"value":1302},{"type":48,"value":3530},{"type":43,"tag":56,"props":3662,"children":3664},{"className":3663},[],[3665],{"type":48,"value":1620},{"type":48,"value":3667}," to focus, then sends the key",{"type":43,"tag":2197,"props":3669,"children":3670},{},[3671,3676,3685],{"type":43,"tag":2242,"props":3672,"children":3673},{},[3674],{"type":48,"value":3675},"Send key to pid",{"type":43,"tag":2242,"props":3677,"children":3678},{},[3679],{"type":43,"tag":56,"props":3680,"children":3682},{"className":3681},[],[3683],{"type":48,"value":3684},"press_key({pid, key, modifiers})",{"type":43,"tag":2242,"props":3686,"children":3687},{},[3688],{"type":48,"value":3689},"no focus change; key goes to pid's current focus",{"type":43,"tag":2197,"props":3691,"children":3692},{},[3693,3698,3716],{"type":43,"tag":2242,"props":3694,"children":3695},{},[3696],{"type":48,"value":3697},"Modifier combo",{"type":43,"tag":2242,"props":3699,"children":3700},{},[3701,3707,3709,3715],{"type":43,"tag":56,"props":3702,"children":3704},{"className":3703},[],[3705],{"type":48,"value":3706},"hotkey({pid, keys})",{"type":48,"value":3708}," (no focus) or ",{"type":43,"tag":56,"props":3710,"children":3712},{"className":3711},[],[3713],{"type":48,"value":3714},"hotkey({pid, x, y, keys})",{"type":48,"value":3519},{"type":43,"tag":2242,"props":3717,"children":3718},{},[3719,3721,3727,3728,3734,3736,3740,3741,3746,3748,3754],{"type":48,"value":3720},"e.g. ",{"type":43,"tag":56,"props":3722,"children":3724},{"className":3723},[],[3725],{"type":48,"value":3726},"[\"cmd\",\"c\"]",{"type":48,"value":639},{"type":43,"tag":56,"props":3729,"children":3731},{"className":3730},[],[3732],{"type":48,"value":3733},"[\"ctrl\",\"c\"]",{"type":48,"value":3735},"; posted per-pid, not HID tap. ",{"type":43,"tag":77,"props":3737,"children":3738},{},[3739],{"type":48,"value":1302},{"type":48,"value":3530},{"type":43,"tag":56,"props":3742,"children":3744},{"className":3743},[],[3745],{"type":48,"value":1620},{"type":48,"value":3747}," to focus a field first, e.g. ",{"type":43,"tag":56,"props":3749,"children":3751},{"className":3750},[],[3752],{"type":48,"value":3753},"[\"cmd\",\"v\"]",{"type":48,"value":3755}," to paste into it",{"type":43,"tag":50,"props":3757,"children":3758},{},[3759,3770],{"type":43,"tag":77,"props":3760,"children":3761},{},[3762,3764,3769],{"type":48,"value":3763},"All keyboard\u002Ftext primitives require ",{"type":43,"tag":56,"props":3765,"children":3767},{"className":3766},[],[3768],{"type":48,"value":503},{"type":48,"value":209},{"type":48,"value":3771}," There is no\nfrontmost-routed variant — every key goes to the named target via\nthe platform's per-pid event-post path, so the driver cannot leak\nkeystrokes into the user's foreground app.",{"type":43,"tag":50,"props":3773,"children":3774},{},[3775,3787],{"type":43,"tag":77,"props":3776,"children":3777},{},[3778,3780,3785],{"type":48,"value":3779},"Why ",{"type":43,"tag":56,"props":3781,"children":3783},{"className":3782},[],[3784],{"type":48,"value":619},{"type":48,"value":3786}," is the primary path:",{"type":48,"value":3788}," works on hidden \u002F\noccluded \u002F off-desktop windows, no focus steal, stable across\nrebuilds, labels tell you what you're clicking. Reach for pixel\ncoordinates only when the accessibility tree can't.",{"type":43,"tag":65,"props":3790,"children":3792},{"id":3791},"cross-platform-parameter-contract",[3793],{"type":48,"value":3794},"Cross-platform parameter contract",{"type":43,"tag":50,"props":3796,"children":3797},{},[3798,3800,3805,3806,3811,3812,3817,3819,3825,3826,3832,3833,3839,3840,3845,3846,3851,3853,3858,3860,3865,3867,3873,3874,3880,3881,3887,3889,3895,3897,3902,3904,3910,3912,3918,3920,3925],{"type":48,"value":3799},"The capture, dispatch, and addressing params — ",{"type":43,"tag":56,"props":3801,"children":3803},{"className":3802},[],[3804],{"type":48,"value":2493},{"type":48,"value":866},{"type":43,"tag":56,"props":3807,"children":3809},{"className":3808},[],[3810],{"type":48,"value":1638},{"type":48,"value":278},{"type":43,"tag":56,"props":3813,"children":3815},{"className":3814},[],[3816],{"type":48,"value":1319},{"type":48,"value":3818}," (deprecated\u002Fignored — see the behavior\nmatrix; still in the schema only so old callers don't error), ",{"type":43,"tag":56,"props":3820,"children":3822},{"className":3821},[],[3823],{"type":48,"value":3824},"scope",{"type":48,"value":866},{"type":43,"tag":56,"props":3827,"children":3829},{"className":3828},[],[3830],{"type":48,"value":3831},"modifier",{"type":48,"value":278},{"type":43,"tag":56,"props":3834,"children":3836},{"className":3835},[],[3837],{"type":48,"value":3838},"button",{"type":48,"value":278},{"type":43,"tag":56,"props":3841,"children":3843},{"className":3842},[],[3844],{"type":48,"value":619},{"type":48,"value":278},{"type":43,"tag":56,"props":3847,"children":3849},{"className":3848},[],[3850],{"type":48,"value":1446},{"type":48,"value":3852}," — are a ",{"type":43,"tag":77,"props":3854,"children":3855},{},[3856],{"type":48,"value":3857},"shared\nschema contract",{"type":48,"value":3859},": identical ",{"type":43,"tag":915,"props":3861,"children":3862},{},[3863],{"type":48,"value":3864},"shape",{"type":48,"value":3866}," (",{"type":43,"tag":56,"props":3868,"children":3870},{"className":3869},[],[3871],{"type":48,"value":3872},"type",{"type":48,"value":114},{"type":43,"tag":56,"props":3875,"children":3877},{"className":3876},[],[3878],{"type":48,"value":3879},"enum",{"type":48,"value":114},{"type":43,"tag":56,"props":3882,"children":3884},{"className":3883},[],[3885],{"type":48,"value":3886},"items",{"type":48,"value":3888},") on macOS,\nWindows, and Linux.\nThey compose from canonical fragments in\n",{"type":43,"tag":56,"props":3890,"children":3892},{"className":3891},[],[3893],{"type":48,"value":3894},"cua-driver-core::tool_schema",{"type":48,"value":3896}," (+ ",{"type":43,"tag":56,"props":3898,"children":3900},{"className":3899},[],[3901],{"type":48,"value":1319},{"type":48,"value":3903},"), and a CI gate\n(",{"type":43,"tag":56,"props":3905,"children":3907},{"className":3906},[],[3908],{"type":48,"value":3909},"schema_consistency_test",{"type":48,"value":3911},") runs every tool's live ",{"type":43,"tag":56,"props":3913,"children":3915},{"className":3914},[],[3916],{"type":48,"value":3917},"tools\u002Flist",{"type":48,"value":3919}," through a\nstructural checker on each platform, so the three surfaces can't\nsilently drift. ",{"type":43,"tag":915,"props":3921,"children":3922},{},[3923],{"type":48,"value":3924},"Contributor note:",{"type":48,"value":3926}," when you add or edit one of these\nshared params on a tool, pull from the fragment — don't re-hand-write the\nJSON, or the gate fails. (Descriptions may legitimately vary per tool;\nthe gate compares shape, not prose.)",{"type":43,"tag":50,"props":3928,"children":3929},{},[3930],{"type":48,"value":3931},"Two consequences for callers:",{"type":43,"tag":85,"props":3933,"children":3934},{},[3935,3971],{"type":43,"tag":89,"props":3936,"children":3937},{},[3938,3948,3950,3955,3957,3962,3964,3970],{"type":43,"tag":77,"props":3939,"children":3940},{},[3941,3946],{"type":43,"tag":56,"props":3942,"children":3944},{"className":3943},[],[3945],{"type":48,"value":2493},{"type":48,"value":3947}," is accepted on every action and cursor tool, on all three\nplatforms.",{"type":48,"value":3949}," It's cursor-wired where the platform glides a cursor and\nschema-accepted everywhere else — so the same ",{"type":43,"tag":56,"props":3951,"children":3953},{"className":3952},[],[3954],{"type":48,"value":2493},{"type":48,"value":3956}," you pass on\nmacOS is no longer ",{"type":43,"tag":915,"props":3958,"children":3959},{},[3960],{"type":48,"value":3961},"rejected",{"type":48,"value":3963}," by Windows\u002FLinux, which previously\nrefused unknown keys via ",{"type":43,"tag":56,"props":3965,"children":3967},{"className":3966},[],[3968],{"type":48,"value":3969},"additionalProperties:false",{"type":48,"value":209},{"type":43,"tag":89,"props":3972,"children":3973},{},[3974,3998,3999,4004,4005,4011,4012,4018,4019,4024,4025,4031,4032,4037,4038,4043,4044,4049,4051,4056,4058,4068,4070,4076,4078,4083,4085,4091,4093,4098,4100,4105,4107,4112,4114,4118,4120,4125,4127,4132],{"type":43,"tag":77,"props":3975,"children":3976},{},[3977,3982,3983,3989,3991,3996],{"type":43,"tag":56,"props":3978,"children":3980},{"className":3979},[],[3981],{"type":48,"value":1638},{"type":48,"value":3866},{"type":43,"tag":56,"props":3984,"children":3986},{"className":3985},[],[3987],{"type":48,"value":3988},"\"background\"",{"type":48,"value":3990}," default \u002F ",{"type":43,"tag":56,"props":3992,"children":3994},{"className":3993},[],[3995],{"type":48,"value":1978},{"type":48,"value":3997},") is on the\nwhole input family",{"type":48,"value":362},{"type":43,"tag":56,"props":4000,"children":4002},{"className":4001},[],[4003],{"type":48,"value":1630},{"type":48,"value":278},{"type":43,"tag":56,"props":4006,"children":4008},{"className":4007},[],[4009],{"type":48,"value":4010},"double_click",{"type":48,"value":278},{"type":43,"tag":56,"props":4013,"children":4015},{"className":4014},[],[4016],{"type":48,"value":4017},"right_click",{"type":48,"value":278},{"type":43,"tag":56,"props":4020,"children":4022},{"className":4021},[],[4023],{"type":48,"value":1775},{"type":48,"value":866},{"type":43,"tag":56,"props":4026,"children":4028},{"className":4027},[],[4029],{"type":48,"value":4030},"scroll",{"type":48,"value":278},{"type":43,"tag":56,"props":4033,"children":4035},{"className":4034},[],[4036],{"type":48,"value":1182},{"type":48,"value":278},{"type":43,"tag":56,"props":4039,"children":4041},{"className":4040},[],[4042],{"type":48,"value":1579},{"type":48,"value":278},{"type":43,"tag":56,"props":4045,"children":4047},{"className":4046},[],[4048],{"type":48,"value":1587},{"type":48,"value":4050}," — uniformly. The\n",{"type":43,"tag":56,"props":4052,"children":4054},{"className":4053},[],[4055],{"type":48,"value":2070},{"type":48,"value":4057}," rung briefly fronts the target, acts, then restores the\nprior frontmost: the explicit last resort when a background attempt\ndidn't land. ",{"type":43,"tag":77,"props":4059,"children":4060},{},[4061,4066],{"type":43,"tag":56,"props":4062,"children":4064},{"className":4063},[],[4065],{"type":48,"value":2070},{"type":48,"value":4067}," is a reaction, never a prediction.",{"type":48,"value":4069}," Always\nfire the ",{"type":43,"tag":56,"props":4071,"children":4073},{"className":4072},[],[4074],{"type":48,"value":4075},"background",{"type":48,"value":4077}," default first and let the driver tell you it\ncan't (a ",{"type":43,"tag":56,"props":4079,"children":4081},{"className":4080},[],[4082],{"type":48,"value":2059},{"type":48,"value":4084}," error or ",{"type":43,"tag":56,"props":4086,"children":4088},{"className":4087},[],[4089],{"type":48,"value":4090},"escalation.recommended == \"foreground\"",{"type":48,"value":4092},") — or observe a verified no-op — ",{"type":43,"tag":915,"props":4094,"children":4095},{},[4096],{"type":48,"value":4097},"before",{"type":48,"value":4099}," you escalate.\nDo ",{"type":43,"tag":77,"props":4101,"children":4102},{},[4103],{"type":48,"value":4104},"not",{"type":48,"value":4106}," reason \"it's a GTK\u002FChromium\u002FElectron app, so background will\ndrop, so I'll front up-front\": the toolkit lists in the tool schemas\nare the ",{"type":43,"tag":915,"props":4108,"children":4109},{},[4110],{"type":48,"value":4111},"driver's",{"type":48,"value":4113}," internal detectors, not a checklist for you to front\non a guess. (Concretely: GIMP's GTK toolbox accepts background pixel\nclicks fine — a preemptive foreground click there just steals the\nuser's focus for nothing.) What each platform's ",{"type":43,"tag":915,"props":4115,"children":4116},{},[4117],{"type":48,"value":4075},{"type":48,"value":4119}," rung can\nactually carry differs (e.g. a Windows background click can't carry\n",{"type":43,"tag":56,"props":4121,"children":4123},{"className":4122},[],[4124],{"type":48,"value":3831},{"type":48,"value":4126}," state — see ",{"type":43,"tag":56,"props":4128,"children":4130},{"className":4129},[],[4131],{"type":48,"value":144},{"type":48,"value":4133},"); the schema is uniform, the\nresidual limits are per-OS.",{"type":43,"tag":50,"props":4135,"children":4136},{},[4137,4142,4143,4148,4150,4156,4158,4163,4164,4170,4171,4177,4179,4185,4187,4192,4194,4199,4201,4206],{"type":43,"tag":77,"props":4138,"children":4139},{},[4140],{"type":48,"value":4141},"Required-set contract.",{"type":48,"value":1143},{"type":43,"tag":56,"props":4144,"children":4146},{"className":4145},[],[4147],{"type":48,"value":1630},{"type":48,"value":4149}," requires nothing (",{"type":43,"tag":56,"props":4151,"children":4153},{"className":4152},[],[4154],{"type":48,"value":4155},"required:[]",{"type":48,"value":4157},"),\n",{"type":43,"tag":56,"props":4159,"children":4161},{"className":4160},[],[4162],{"type":48,"value":4030},{"type":48,"value":497},{"type":43,"tag":56,"props":4165,"children":4167},{"className":4166},[],[4168],{"type":48,"value":4169},"[\"direction\"]",{"type":48,"value":278},{"type":43,"tag":56,"props":4172,"children":4174},{"className":4173},[],[4175],{"type":48,"value":4176},"zoom",{"type":48,"value":4178}," requires\n",{"type":43,"tag":56,"props":4180,"children":4182},{"className":4181},[],[4183],{"type":48,"value":4184},"[\"window_id\",\"x1\",\"y1\",\"x2\",\"y2\"]",{"type":48,"value":4186}," — same on every platform. ",{"type":43,"tag":56,"props":4188,"children":4190},{"className":4189},[],[4191],{"type":48,"value":503},{"type":48,"value":4193}," is\n",{"type":43,"tag":77,"props":4195,"children":4196},{},[4197],{"type":48,"value":4198},"conditionally",{"type":48,"value":4200}," required (needed unless a windowless desktop-scope\ncall) and validated in code with a clear error, NOT pinned in the schema\n— so omitting ",{"type":43,"tag":56,"props":4202,"children":4204},{"className":4203},[],[4205],{"type":48,"value":503},{"type":48,"value":4207}," for a desktop-scope action is no longer\nschema-rejected.",{"type":43,"tag":50,"props":4209,"children":4210},{},[4211,4213,4219,4221,4227],{"type":48,"value":4212},"Genuinely platform-specific params stay OUT of the shared contract by\ndesign (launch-app identifiers, the Windows-only ",{"type":43,"tag":56,"props":4214,"children":4216},{"className":4215},[],[4217],{"type":48,"value":4218},"debug_window_info",{"type":48,"value":4220},", the\nmacOS-only ",{"type":43,"tag":56,"props":4222,"children":4224},{"className":4223},[],[4225],{"type":48,"value":4226},"check_permissions.prompt",{"type":48,"value":4228},"). The per-OS files list the\nresiduals that matter when you drive on that platform.",{"type":43,"tag":65,"props":4230,"children":4232},{"id":4231},"pixel-coordinate-clicks",[4233],{"type":48,"value":4234},"Pixel-coordinate clicks",{"type":43,"tag":50,"props":4236,"children":4237},{},[4238,4240,4245,4247,4252,4254,4259,4261,4266,4268,4274],{"type":48,"value":4239},"The pixel path (",{"type":43,"tag":56,"props":4241,"children":4243},{"className":4242},[],[4244],{"type":48,"value":3428},{"type":48,"value":4246},") is for surfaces the\naccessibility tree doesn't reach — canvases, video players, WebGL,\ncustom-drawn controls. Coords are ",{"type":43,"tag":77,"props":4248,"children":4249},{},[4250],{"type":48,"value":4251},"window-local screenshot pixels",{"type":48,"value":4253},"\n(same space as the PNG ",{"type":43,"tag":56,"props":4255,"children":4257},{"className":4256},[],[4258],{"type":48,"value":1131},{"type":48,"value":4260}," returns). Top-left origin,\ny-down. The driver handles screen-point conversion internally.\nPassing ",{"type":43,"tag":56,"props":4262,"children":4264},{"className":4263},[],[4265],{"type":48,"value":511},{"type":48,"value":4267}," alongside ",{"type":43,"tag":56,"props":4269,"children":4271},{"className":4270},[],[4272],{"type":48,"value":4273},"x, y",{"type":48,"value":4275}," is optional but recommended —\nit pins the coordinate conversion to the window whose screenshot\nproduced the pixel.",{"type":43,"tag":50,"props":4277,"children":4278},{},[4279,4281,4286,4288,4293,4294,4300,4302,4307],{"type":48,"value":4280},"PNGs returned by ",{"type":43,"tag":56,"props":4282,"children":4284},{"className":4283},[],[4285],{"type":48,"value":1131},{"type":48,"value":4287}," are capped at ",{"type":43,"tag":77,"props":4289,"children":4290},{},[4291],{"type":48,"value":4292},"1568 px long-side\nby default",{"type":48,"value":3866},{"type":43,"tag":56,"props":4295,"children":4297},{"className":4296},[],[4298],{"type":48,"value":4299},"max_image_dimension",{"type":48,"value":4301}," config), matching Anthropic's\nmultimodal-vision downsampling limit. The image the model reasons\nover and the image the click tool's coordinate system lives in are\nthe ",{"type":43,"tag":77,"props":4303,"children":4304},{},[4305],{"type":48,"value":4306},"same resolution",{"type":48,"value":4308}," — just look at the PNG, pick a pixel, click\nat that pixel. No scaling math.",{"type":43,"tag":50,"props":4310,"children":4311},{},[4312,4314,4320],{"type":48,"value":4313},"This is the default because the mismatch between \"rendered\nthumbnail\" and \"native PNG\" was a recurring coord-estimation\nfootgun. If you opt out (explicit ",{"type":43,"tag":56,"props":4315,"children":4317},{"className":4316},[],[4318],{"type":48,"value":4319},"max_image_dimension=0",{"type":48,"value":4321}," for\npixel-perfect verification flows), the old rule applies: don't\neyeball coords from whatever your client renders — it may be\n2-4× smaller than the PNG on disk, and a 2% error in thumbnail\nspace becomes ~80 px in the real image.",{"type":43,"tag":50,"props":4323,"children":4324},{},[4325],{"type":48,"value":4326},"For precise targeting on small \u002F dense UIs:",{"type":43,"tag":4328,"props":4329,"children":4330},"ol",{},[4331,4362,4367],{"type":43,"tag":89,"props":4332,"children":4333},{},[4334,4339,4341,4346,4347,4353,4355,4361],{"type":43,"tag":56,"props":4335,"children":4337},{"className":4336},[],[4338],{"type":48,"value":2713},{"type":48,"value":4340}," → image capped at 1568\nlong-side plus ",{"type":43,"tag":56,"props":4342,"children":4344},{"className":4343},[],[4345],{"type":48,"value":2896},{"type":48,"value":639},{"type":43,"tag":56,"props":4348,"children":4350},{"className":4349},[],[4351],{"type":48,"value":4352},"screenshot_height",{"type":48,"value":4354},". Write to\ndisk via ",{"type":43,"tag":56,"props":4356,"children":4358},{"className":4357},[],[4359],{"type":48,"value":4360},"--screenshot-out-file \u003Cpath>",{"type":48,"value":209},{"type":43,"tag":89,"props":4363,"children":4364},{},[4365],{"type":48,"value":4366},"Look at the PNG. Since it matches what you see, pick the target\npixel directly.",{"type":43,"tag":89,"props":4368,"children":4369},{},[4370,4372,4376],{"type":48,"value":4371},"When precision matters, draw a crosshair on the image (do\n",{"type":43,"tag":77,"props":4373,"children":4374},{},[4375],{"type":48,"value":4104},{"type":48,"value":4377}," crop — cropping loses the coordinate system) and verify\nbefore clicking:",{"type":43,"tag":435,"props":4379,"children":4383},{"className":4380,"code":4381,"language":4382,"meta":440,"style":440},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from PIL import Image, ImageDraw\nimg = Image.open('\u002Ftmp\u002Fshot.png')\ndraw = ImageDraw.Draw(img)\nx, y = \u003Cyour_coordinate>\nr = 18\ndraw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)\ndraw.line([x-30, y, x+30, y], fill='red', width=3)\ndraw.line([x, y-30, x, y+30], fill='red', width=3)\nimg.save('\u002Ftmp\u002Fshot_annotated.png')\n","python",[4384],{"type":43,"tag":56,"props":4385,"children":4386},{"__ignoreMap":440},[4387,4395,4403,4411,4419,4427,4435,4444,4453],{"type":43,"tag":446,"props":4388,"children":4389},{"class":448,"line":449},[4390],{"type":43,"tag":446,"props":4391,"children":4392},{},[4393],{"type":48,"value":4394},"from PIL import Image, ImageDraw\n",{"type":43,"tag":446,"props":4396,"children":4397},{"class":448,"line":731},[4398],{"type":43,"tag":446,"props":4399,"children":4400},{},[4401],{"type":48,"value":4402},"img = Image.open('\u002Ftmp\u002Fshot.png')\n",{"type":43,"tag":446,"props":4404,"children":4405},{"class":448,"line":760},[4406],{"type":43,"tag":446,"props":4407,"children":4408},{},[4409],{"type":48,"value":4410},"draw = ImageDraw.Draw(img)\n",{"type":43,"tag":446,"props":4412,"children":4413},{"class":448,"line":769},[4414],{"type":43,"tag":446,"props":4415,"children":4416},{},[4417],{"type":48,"value":4418},"x, y = \u003Cyour_coordinate>\n",{"type":43,"tag":446,"props":4420,"children":4421},{"class":448,"line":795},[4422],{"type":43,"tag":446,"props":4423,"children":4424},{},[4425],{"type":48,"value":4426},"r = 18\n",{"type":43,"tag":446,"props":4428,"children":4429},{"class":448,"line":821},[4430],{"type":43,"tag":446,"props":4431,"children":4432},{},[4433],{"type":48,"value":4434},"draw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)\n",{"type":43,"tag":446,"props":4436,"children":4438},{"class":448,"line":4437},7,[4439],{"type":43,"tag":446,"props":4440,"children":4441},{},[4442],{"type":48,"value":4443},"draw.line([x-30, y, x+30, y], fill='red', width=3)\n",{"type":43,"tag":446,"props":4445,"children":4447},{"class":448,"line":4446},8,[4448],{"type":43,"tag":446,"props":4449,"children":4450},{},[4451],{"type":48,"value":4452},"draw.line([x, y-30, x, y+30], fill='red', width=3)\n",{"type":43,"tag":446,"props":4454,"children":4456},{"class":448,"line":4455},9,[4457],{"type":43,"tag":446,"props":4458,"children":4459},{},[4460],{"type":48,"value":4461},"img.save('\u002Ftmp\u002Fshot_annotated.png')\n",{"type":43,"tag":4328,"props":4463,"children":4464},{"start":769},[4465],{"type":43,"tag":89,"props":4466,"children":4467},{},[4468],{"type":48,"value":4469},"Only dispatch the click after the user (or your own re-read of\nthe annotated image) confirms the crosshair is on target.",{"type":43,"tag":50,"props":4471,"children":4472},{},[4473],{"type":48,"value":4474},"Addressing variants:",{"type":43,"tag":85,"props":4476,"children":4477},{},[4478,4488,4499,4517],{"type":43,"tag":89,"props":4479,"children":4480},{},[4481,4486],{"type":43,"tag":56,"props":4482,"children":4484},{"className":4483},[],[4485],{"type":48,"value":3428},{"type":48,"value":4487}," — single left-click.",{"type":43,"tag":89,"props":4489,"children":4490},{},[4491,4497],{"type":43,"tag":56,"props":4492,"children":4494},{"className":4493},[],[4495],{"type":48,"value":4496},"click({pid, x, y, count: 2})",{"type":48,"value":4498}," — double-click.",{"type":43,"tag":89,"props":4500,"children":4501},{},[4502,4508,4510,4516],{"type":43,"tag":56,"props":4503,"children":4505},{"className":4504},[],[4506],{"type":48,"value":4507},"click({pid, x, y, modifier: [\"cmd\"\\|\"ctrl\"]})",{"type":48,"value":4509}," — modifier click.\nAccepts any subset of ",{"type":43,"tag":56,"props":4511,"children":4513},{"className":4512},[],[4514],{"type":48,"value":4515},"cmd\u002Fshift\u002Foption\u002Falt\u002Fctrl",{"type":48,"value":209},{"type":43,"tag":89,"props":4518,"children":4519},{},[4520,4526,4528,4533],{"type":43,"tag":56,"props":4521,"children":4523},{"className":4522},[],[4524],{"type":48,"value":4525},"right_click({pid, x, y})",{"type":48,"value":4527}," — also takes ",{"type":43,"tag":56,"props":4529,"children":4531},{"className":4530},[],[4532],{"type":48,"value":3831},{"type":48,"value":209},{"type":43,"tag":50,"props":4535,"children":4536},{},[4537,4539,4545],{"type":48,"value":4538},"The pixel path animates the agent cursor overlay but never warps\nthe real cursor (the per-pid event paths the driver uses on macOS\nand Windows route around HID synthesis). If the pid has no on-screen\nwindow the call errors with ",{"type":43,"tag":56,"props":4540,"children":4542},{"className":4541},[],[4543],{"type":48,"value":4544},"pid X has no on-screen window",{"type":48,"value":4546}," — you\nneed a visible window to anchor the conversion. Dispatch details\n(SkyLight on macOS, layered UIA+PostMessage on Windows) are in the\nper-OS companion files.",{"type":43,"tag":65,"props":4548,"children":4550},{"id":4549},"web-rendered-apps-browsers-electron-tauri",[4551],{"type":48,"value":4552},"Web-rendered apps (browsers, Electron, Tauri)",{"type":43,"tag":50,"props":4554,"children":4555},{},[4556,4558,4566],{"type":48,"value":4557},"For Chrome \u002F Edge \u002F Brave \u002F Arc \u002F Safari, Electron apps (Slack,\nVSCode, Notion, Discord), and Tauri apps — see ",{"type":43,"tag":77,"props":4559,"children":4560},{},[4561],{"type":43,"tag":56,"props":4562,"children":4564},{"className":4563},[],[4565],{"type":48,"value":180},{"type":48,"value":209},{"type":43,"tag":50,"props":4568,"children":4569},{},[4570,4572,4577,4579,4583,4585,4590],{"type":48,"value":4571},"Covers: sparse accessibility tree population (retry-once pattern for\nChromium), URL navigation via omnibox suggestions, the ",{"type":43,"tag":56,"props":4573,"children":4575},{"className":4574},[],[4576],{"type":48,"value":188},{"type":48,"value":4578},"\nworkaround for keyboard commits on ",{"type":43,"tag":77,"props":4580,"children":4581},{},[4582],{"type":48,"value":2298},{"type":48,"value":4584}," windows (Return\nsilently no-ops — symptom is a system bell; use ",{"type":43,"tag":56,"props":4586,"children":4588},{"className":4587},[],[4589],{"type":48,"value":188},{"type":48,"value":4591}," or click\na clickable equivalent), scrolling via synthetic PageUp\u002FDown\nkeystrokes, in-page clicks, and typing into web inputs.",{"type":43,"tag":50,"props":4593,"children":4594},{},[4595,4597,4602,4604,4610,4612,4617,4619,4625,4626,4632,4634,4640,4642,4648,4650,4656],{"type":48,"value":4596},"Browser JS primitives are now ",{"type":43,"tag":77,"props":4598,"children":4599},{},[4600],{"type":48,"value":4601},"cross-platform",{"type":48,"value":4603}," via the ",{"type":43,"tag":56,"props":4605,"children":4607},{"className":4606},[],[4608],{"type":48,"value":4609},"page",{"type":48,"value":4611}," tool —\nmacOS uses Apple Events for Chrome\u002FBrave\u002FEdge\u002FSafari + CDP for Electron\n(see ",{"type":43,"tag":56,"props":4613,"children":4615},{"className":4614},[],[4616],{"type":48,"value":104},{"type":48,"value":4618},"); Windows + Linux use UIA \u002F AT-SPI for ",{"type":43,"tag":56,"props":4620,"children":4622},{"className":4621},[],[4623],{"type":48,"value":4624},"get_text",{"type":48,"value":980},{"type":43,"tag":56,"props":4627,"children":4629},{"className":4628},[],[4630],{"type":48,"value":4631},"query_dom",{"type":48,"value":4633}," and the shared CDP client for ",{"type":43,"tag":56,"props":4635,"children":4637},{"className":4636},[],[4638],{"type":48,"value":4639},"execute_javascript",{"type":48,"value":4641}," (browser\nmust be launched with ",{"type":43,"tag":56,"props":4643,"children":4645},{"className":4644},[],[4646],{"type":48,"value":4647},"--remote-debugging-port=N",{"type":48,"value":4649}," and the port exported\nas ",{"type":43,"tag":56,"props":4651,"children":4653},{"className":4652},[],[4654],{"type":48,"value":4655},"CUA_DRIVER_CDP_PORT",{"type":48,"value":2919},{"type":43,"tag":65,"props":4658,"children":4660},{"id":4659},"re-snapshot-and-verify-mandatory",[4661],{"type":48,"value":4662},"Re-snapshot and verify — mandatory",{"type":43,"tag":50,"props":4664,"children":4665},{},[4666,4671,4673,4678],{"type":43,"tag":77,"props":4667,"children":4668},{},[4669],{"type":48,"value":4670},"Always",{"type":48,"value":4672}," call ",{"type":43,"tag":56,"props":4674,"children":4676},{"className":4675},[],[4677],{"type":48,"value":2713},{"type":48,"value":4679}," after the\naction. This isn't optional verification — it's the second half of\nthe snapshot invariant.",{"type":43,"tag":50,"props":4681,"children":4682},{},[4683,4685,4690],{"type":48,"value":4684},"Check the tree diff: a changed value, a new element, a new window,\nor the disappearance of the thing you just clicked (menus collapse\nafter selection, buttons may become disabled, etc.). The re-snapshot\ngives you both the tree and the screenshot, so you cross-check the tree\ndiff against the pixels in one call — and when you're only confirming a\ntree change, ",{"type":43,"tag":56,"props":4686,"children":4688},{"className":4687},[],[4689],{"type":48,"value":1280},{"type":48,"value":4691}," skips the grab.",{"type":43,"tag":50,"props":4693,"children":4694},{},[4695,4697,4701,4703,4709,4711,4716,4718,4724,4726,4731,4733,4738,4740,4745,4747,4752],{"type":48,"value":4696},"Switch to an ",{"type":43,"tag":77,"props":4698,"children":4699},{},[4700],{"type":48,"value":1478},{"type":48,"value":4702}," only on a real signal: the action\nresponse carried ",{"type":43,"tag":56,"props":4704,"children":4706},{"className":4705},[],[4707],{"type":48,"value":4708},"effect:\"suspected_noop\"",{"type":48,"value":4710},", the re-snapshot came back\n",{"type":43,"tag":56,"props":4712,"children":4714},{"className":4713},[],[4715],{"type":48,"value":2157},{"type":48,"value":4717}," (empty tree → non-AX surface), the tree looks\nunchanged\u002Funreadable or disagrees with the screenshot, or\n",{"type":43,"tag":56,"props":4719,"children":4721},{"className":4720},[],[4722],{"type":48,"value":4723},"escalation.recommended",{"type":48,"value":4725}," points you there (",{"type":43,"tag":56,"props":4727,"children":4729},{"className":4728},[],[4730],{"type":48,"value":1302},{"type":48,"value":4732},"). That's the\nverify-then-escalate ladder in the behavior-matrix section. If the tree\nis unchanged AND the screenshot confirms nothing moved, the action\nlikely failed silently — ",{"type":43,"tag":77,"props":4734,"children":4735},{},[4736],{"type":48,"value":4737},"tell the user what you attempted and what\nyou observed",{"type":48,"value":4739},", don't paper over with \"done\" language (and consider\n",{"type":43,"tag":56,"props":4741,"children":4743},{"className":4742},[],[4744],{"type":48,"value":1731},{"type":48,"value":4746}," when ",{"type":43,"tag":56,"props":4748,"children":4750},{"className":4749},[],[4751],{"type":48,"value":4090},{"type":48,"value":4753},"). Agents that skip this step report success on\nsilently-dropped actions — the single most common failure mode.",{"type":43,"tag":65,"props":4755,"children":4757},{"id":4756},"recording-trajectories",[4758],{"type":48,"value":4759},"Recording trajectories",{"type":43,"tag":50,"props":4761,"children":4762},{},[4763,4765,4770,4772,4778,4779,4785,4787,4793,4795,4801],{"type":48,"value":4764},"Session-scoped action recording + replay, for demos, regressions,\nand training data. Only invoke when the user explicitly asks to\nrecord a session — the skill does not auto-enable this. CLI surface:\n",{"type":43,"tag":56,"props":4766,"children":4768},{"className":4767},[],[4769],{"type":48,"value":670},{"type":48,"value":4771},"; raw tools:\n",{"type":43,"tag":56,"props":4773,"children":4775},{"className":4774},[],[4776],{"type":48,"value":4777},"start_recording",{"type":48,"value":639},{"type":43,"tag":56,"props":4780,"children":4782},{"className":4781},[],[4783],{"type":48,"value":4784},"stop_recording",{"type":48,"value":4786},". Video capture (main display →\n",{"type":43,"tag":56,"props":4788,"children":4790},{"className":4789},[],[4791],{"type":48,"value":4792},"recording.mp4",{"type":48,"value":4794},") is on by default; pass ",{"type":43,"tag":56,"props":4796,"children":4798},{"className":4797},[],[4799],{"type":48,"value":4800},"record_video: false",{"type":48,"value":4802}," to opt out.",{"type":43,"tag":50,"props":4804,"children":4805},{},[4806,4808,4816,4818,4823],{"type":48,"value":4807},"See ",{"type":43,"tag":77,"props":4809,"children":4810},{},[4811],{"type":43,"tag":56,"props":4812,"children":4814},{"className":4813},[],[4815],{"type":48,"value":199},{"type":48,"value":4817}," for the full flow: enable\u002Fdisable, turn folder\ncontents, replay via ",{"type":43,"tag":56,"props":4819,"children":4821},{"className":4820},[],[4822],{"type":48,"value":207},{"type":48,"value":4824},", and the element_index\ndoesn't-survive-across-sessions caveat.",{"type":43,"tag":65,"props":4826,"children":4828},{"id":4827},"common-error-patterns-cross-platform",[4829],{"type":48,"value":4830},"Common error patterns (cross-platform)",{"type":43,"tag":2189,"props":4832,"children":4833},{},[4834,4855],{"type":43,"tag":2193,"props":4835,"children":4836},{},[4837],{"type":43,"tag":2197,"props":4838,"children":4839},{},[4840,4845,4850],{"type":43,"tag":2201,"props":4841,"children":4842},{},[4843],{"type":48,"value":4844},"Error text",{"type":43,"tag":2201,"props":4846,"children":4847},{},[4848],{"type":48,"value":4849},"Meaning",{"type":43,"tag":2201,"props":4851,"children":4852},{},[4853],{"type":48,"value":4854},"Fix",{"type":43,"tag":2235,"props":4856,"children":4857},{},[4858,4901,4930,4960,5018],{"type":43,"tag":2197,"props":4859,"children":4860},{},[4861,4870,4889],{"type":43,"tag":2242,"props":4862,"children":4863},{},[4864],{"type":43,"tag":56,"props":4865,"children":4867},{"className":4866},[],[4868],{"type":48,"value":4869},"No cached AX state for pid X window_id W",{"type":43,"tag":2242,"props":4871,"children":4872},{},[4873,4875,4880,4882,4887],{"type":48,"value":4874},"You either skipped ",{"type":43,"tag":56,"props":4876,"children":4878},{"className":4877},[],[4879],{"type":48,"value":1131},{"type":48,"value":4881}," this turn, or passed a different ",{"type":43,"tag":56,"props":4883,"children":4885},{"className":4884},[],[4886],{"type":48,"value":511},{"type":48,"value":4888}," to the click than the one the snapshot cached against",{"type":43,"tag":2242,"props":4890,"children":4891},{},[4892,4893,4899],{"type":48,"value":2707},{"type":43,"tag":56,"props":4894,"children":4896},{"className":4895},[],[4897],{"type":48,"value":4898},"get_window_state({pid: X, window_id: W})",{"type":48,"value":4900}," first — the same window_id you intend to click in",{"type":43,"tag":2197,"props":4902,"children":4903},{},[4904,4913,4918],{"type":43,"tag":2242,"props":4905,"children":4906},{},[4907],{"type":43,"tag":56,"props":4908,"children":4910},{"className":4909},[],[4911],{"type":48,"value":4912},"Invalid element_index N for pid X window_id W",{"type":43,"tag":2242,"props":4914,"children":4915},{},[4916],{"type":48,"value":4917},"Index is stale or out of range",{"type":43,"tag":2242,"props":4919,"children":4920},{},[4921,4923,4928],{"type":48,"value":4922},"Re-run ",{"type":43,"tag":56,"props":4924,"children":4926},{"className":4925},[],[4927],{"type":48,"value":1131},{"type":48,"value":4929}," with the same window_id, pick a fresh index from the new tree",{"type":43,"tag":2197,"props":4931,"children":4932},{},[4933,4942,4947],{"type":43,"tag":2242,"props":4934,"children":4935},{},[4936],{"type":43,"tag":56,"props":4937,"children":4939},{"className":4938},[],[4940],{"type":48,"value":4941},"window_id W belongs to pid P, not …",{"type":43,"tag":2242,"props":4943,"children":4944},{},[4945],{"type":48,"value":4946},"Passed a window_id that's owned by a different process",{"type":43,"tag":2242,"props":4948,"children":4949},{},[4950,4952,4958],{"type":48,"value":4951},"Use ",{"type":43,"tag":56,"props":4953,"children":4955},{"className":4954},[],[4956],{"type":48,"value":4957},"list_windows({pid: X})",{"type":48,"value":4959}," to enumerate this pid's own windows",{"type":43,"tag":2197,"props":4961,"children":4962},{},[4963,4979,4984],{"type":43,"tag":2242,"props":4964,"children":4965},{},[4966,4972,4973],{"type":43,"tag":56,"props":4967,"children":4969},{"className":4968},[],[4970],{"type":48,"value":4971},"AX action … failed with code …",{"type":48,"value":639},{"type":43,"tag":56,"props":4974,"children":4976},{"className":4975},[],[4977],{"type":48,"value":4978},"UIA invoke failed",{"type":43,"tag":2242,"props":4980,"children":4981},{},[4982],{"type":48,"value":4983},"Element doesn't support the default action",{"type":43,"tag":2242,"props":4985,"children":4986},{},[4987,4989,4995,4996,5002,5003,5009,5010,5016],{"type":48,"value":4988},"Try ",{"type":43,"tag":56,"props":4990,"children":4992},{"className":4991},[],[4993],{"type":48,"value":4994},"show_menu",{"type":48,"value":278},{"type":43,"tag":56,"props":4997,"children":4999},{"className":4998},[],[5000],{"type":48,"value":5001},"confirm",{"type":48,"value":278},{"type":43,"tag":56,"props":5004,"children":5006},{"className":5005},[],[5007],{"type":48,"value":5008},"cancel",{"type":48,"value":278},{"type":43,"tag":56,"props":5011,"children":5013},{"className":5012},[],[5014],{"type":48,"value":5015},"pick",{"type":48,"value":5017},", or fall through to a pixel click on the element's center",{"type":43,"tag":2197,"props":5019,"children":5020},{},[5021,5030,5042],{"type":43,"tag":2242,"props":5022,"children":5023},{},[5024],{"type":43,"tag":56,"props":5025,"children":5027},{"className":5026},[],[5028],{"type":48,"value":5029},"The user doesn't want to proceed with this tool use. The tool use was rejected …",{"type":43,"tag":2242,"props":5031,"children":5032},{},[5033,5035,5040],{"type":48,"value":5034},"The harness uses this ",{"type":43,"tag":915,"props":5036,"children":5037},{},[5038],{"type":48,"value":5039},"exact",{"type":48,"value":5041}," string for BOTH a permission-prompt denial AND a manual interrupt (Esc \u002F stop) of a running tool — they are indistinguishable from the tool result",{"type":43,"tag":2242,"props":5043,"children":5044},{},[5045],{"type":48,"value":5046},"Treat as \"tool canceled, no result, await the user.\" Do NOT paraphrase (\"you stopped me\") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed",{"type":43,"tag":50,"props":5048,"children":5049},{},[5050],{"type":48,"value":5051},"Platform-specific errors (TCC dialogs on macOS, Session 0 \u002F UAC\nprompts on Windows, AT-SPI bus issues on Linux) live in their\nrespective companion files.",{"type":43,"tag":65,"props":5053,"children":5055},{"id":5054},"things-to-avoid",[5056],{"type":48,"value":5057},"Things to avoid",{"type":43,"tag":85,"props":5059,"children":5060},{},[5061,5078,5141,5163,5172],{"type":43,"tag":89,"props":5062,"children":5063},{},[5064,5069,5071,5076],{"type":43,"tag":77,"props":5065,"children":5066},{},[5067],{"type":48,"value":5068},"Never",{"type":48,"value":5070}," reuse an ",{"type":43,"tag":56,"props":5072,"children":5074},{"className":5073},[],[5075],{"type":48,"value":619},{"type":48,"value":5077}," across a re-snapshot of the same pid.",{"type":43,"tag":89,"props":5079,"children":5080},{},[5081,5086,5088,5093,5095,5099,5101,5105,5107,5112,5114,5119,5121,5126,5127,5132,5134,5139],{"type":43,"tag":77,"props":5082,"children":5083},{},[5084],{"type":48,"value":5085},"Don't conflate the two addressing modes.",{"type":48,"value":5087}," The tree gives you\n",{"type":43,"tag":56,"props":5089,"children":5091},{"className":5090},[],[5092],{"type":48,"value":619},{"type":48,"value":5094}," handles; the screenshot (same call) gives you the\npixel frame. An ",{"type":43,"tag":77,"props":5096,"children":5097},{},[5098],{"type":48,"value":1432},{"type":48,"value":5100}," addresses by index, an\n",{"type":43,"tag":77,"props":5102,"children":5103},{},[5104],{"type":48,"value":1478},{"type":48,"value":5106}," by ",{"type":43,"tag":56,"props":5108,"children":5110},{"className":5109},[],[5111],{"type":48,"value":1537},{"type":48,"value":5113},". Default to ",{"type":43,"tag":56,"props":5115,"children":5117},{"className":5116},[],[5118],{"type":48,"value":619},{"type":48,"value":5120}," and only\ndo a px action on a real signal (",{"type":43,"tag":56,"props":5122,"children":5124},{"className":5123},[],[5125],{"type":48,"value":2150},{"type":48,"value":639},{"type":43,"tag":56,"props":5128,"children":5130},{"className":5129},[],[5131],{"type":48,"value":2157},{"type":48,"value":5133}," \u002F\nrepeated labels \u002F tree-disagrees-with-pixels). Don't pass an\n",{"type":43,"tag":56,"props":5135,"children":5137},{"className":5136},[],[5138],{"type":48,"value":619},{"type":48,"value":5140}," you read off the screenshot, and don't pixel-click a\ncoordinate you computed from the tree's (possibly lying) frame\nwithout checking it against the image.",{"type":43,"tag":89,"props":5142,"children":5143},{},[5144,5149,5150,5155,5157,5161],{"type":43,"tag":77,"props":5145,"children":5146},{},[5147],{"type":48,"value":5148},"Prefer accessibility actions over pixels.",{"type":48,"value":1143},{"type":43,"tag":56,"props":5151,"children":5153},{"className":5152},[],[5154],{"type":48,"value":3428},{"type":48,"value":5156},"\nworks for canvas \u002F WebView regions, but it lands blindly on raw\ncoordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,\ntoolbar items, keyboard shortcuts) before dropping to coordinates.\n(The AX path does ",{"type":43,"tag":77,"props":5158,"children":5159},{},[5160],{"type":48,"value":4104},{"type":48,"value":5162}," skip the agent-cursor overlay — it seeds and\npulses the session cursor and draws a focus rect on the targeted\nelement; it just doesn't play a long glide on the very first action.\nSee \"Agent cursor overlay\" for the demo-recording caveat.)",{"type":43,"tag":89,"props":5164,"children":5165},{},[5166,5170],{"type":43,"tag":77,"props":5167,"children":5168},{},[5169],{"type":48,"value":5068},{"type":48,"value":5171}," drive destructive actions (delete files, close unsaved\ndocuments, send messages, submit forms) without explicit user\nintent for that specific destructive step.",{"type":43,"tag":89,"props":5173,"children":5174},{},[5175,5179],{"type":43,"tag":77,"props":5176,"children":5177},{},[5178],{"type":48,"value":5068},{"type":48,"value":5180}," launch apps autonomously; confirm with the user first\nunless their original request clearly implies the launch.",{"type":43,"tag":65,"props":5182,"children":5184},{"id":5183},"example-end-to-end-task",[5185],{"type":48,"value":5186},"Example end-to-end task",{"type":43,"tag":50,"props":5188,"children":5189},{},[5190,5195],{"type":43,"tag":77,"props":5191,"children":5192},{},[5193],{"type":48,"value":5194},"User:",{"type":48,"value":5196}," \"Open the Downloads folder in the system file manager.\"",{"type":43,"tag":4328,"props":5198,"children":5199},{},[5200,5227,5237],{"type":43,"tag":89,"props":5201,"children":5202},{},[5203,5209,5211,5217,5219,5225],{"type":43,"tag":56,"props":5204,"children":5206},{"className":5205},[],[5207],{"type":48,"value":5208},"launch_app({bundle_id: \"com.apple.finder\", urls: [\"~\u002FDownloads\"]})",{"type":48,"value":5210},"\non macOS, or ",{"type":43,"tag":56,"props":5212,"children":5214},{"className":5213},[],[5215],{"type":48,"value":5216},"launch_app({name: \"explorer\", args: [\"%USERPROFILE%\\\\Downloads\"]})",{"type":48,"value":5218},"\non Windows. Returns ",{"type":43,"tag":56,"props":5220,"children":5222},{"className":5221},[],[5223],{"type":48,"value":5224},"{pid, windows: [{window_id, title, ...}]}",{"type":48,"value":5226},".\nIdempotent launch; the driver opens a hidden window via the\nplatform's launch primitive — zero activation, no focus steal.",{"type":43,"tag":89,"props":5228,"children":5229},{},[5230,5235],{"type":43,"tag":56,"props":5231,"children":5233},{"className":5232},[],[5234],{"type":48,"value":2713},{"type":48,"value":5236}," → verify the expected window\ntitle is present with a populated tree (sidebar, list view, files).",{"type":43,"tag":89,"props":5238,"children":5239},{},[5240],{"type":48,"value":5241},"Done.",{"type":43,"tag":50,"props":5243,"children":5244},{},[5245],{"type":48,"value":5246},"Platform-specific examples and edge cases (Finder menu navigation,\nExplorer ribbon, GNOME Files) live in the per-OS companion files.",{"type":43,"tag":5248,"props":5249,"children":5250},"style",{},[5251],{"type":48,"value":5252},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5254,"total":5423},[5255,5268,5275,5291,5307,5321,5337,5351,5364,5378,5390,5405],{"slug":5256,"name":5256,"fn":5257,"description":5258,"org":5259,"tags":5260,"stars":26,"repoUrl":27,"updatedAt":5267},"batch","execute parallel batch operations on files","Execute batch operations on multiple files in parallel. Automatically discovers files, splits into chunks, and processes with parallel worker agents. Use `\u002Fbatch` followed by operation and file pattern.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5261,5264],{"name":5262,"slug":5263,"type":16},"Automation","automation",{"name":5265,"slug":5266,"type":16},"Concurrency","concurrency","2026-07-20T05:58:45.650067",{"slug":4,"name":4,"fn":5,"description":6,"org":5269,"tags":5270,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5271,5272,5273,5274],{"name":14,"slug":15,"type":16},{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":5276,"name":5276,"fn":5277,"description":5278,"org":5279,"tags":5280,"stars":26,"repoUrl":27,"updatedAt":5290},"dataviz","design and validate data visualizations","Design guidance for charts, graphs, dashboards, maps, and data visualizations, including a local palette validator.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5281,5284,5287],{"name":5282,"slug":5283,"type":16},"Charts","charts",{"name":5285,"slug":5286,"type":16},"Dashboards","dashboards",{"name":5288,"slug":5289,"type":16},"Data Visualization","data-visualization","2026-07-16T05:59:28.31755",{"slug":5292,"name":5292,"fn":5293,"description":5294,"org":5295,"tags":5296,"stars":26,"repoUrl":27,"updatedAt":5306},"extension-creator","create Qwen Code extensions","Create, scaffold, customize, validate, and locally test Qwen Code extensions. Use when the user wants a new Qwen Code extension, needs help choosing an extension template, wants to add QWEN.md context, commands, skills, agents, MCP servers, settings, hooks, channels, or LSP servers, or asks how to link and test an extension locally. Invoke with `\u002Fextension-creator` followed by an extension path and optional template name.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5297,5300,5303],{"name":5298,"slug":5299,"type":16},"Coding","coding",{"name":5301,"slug":5302,"type":16},"Documentation","documentation",{"name":5304,"slug":5305,"type":16},"Plugin Development","plugin-development","2026-07-16T05:59:24.818967",{"slug":5308,"name":5308,"fn":5309,"description":5310,"org":5311,"tags":5312,"stars":26,"repoUrl":27,"updatedAt":5320},"loop","run scheduled or self-paced prompt loops","Create a loop that runs a prompt now and follows up either on a fixed schedule or through self-paced wakeups. Usage - \u002Floop check the build, \u002Floop 5m check the build, \u002Floop check the PR every 30m. \u002Floop list to show jobs, \u002Floop clear to cancel all.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5313,5314,5317],{"name":5262,"slug":5263,"type":16},{"name":5315,"slug":5316,"type":16},"Productivity","productivity",{"name":5318,"slug":5319,"type":16},"Scheduling","scheduling","2026-07-16T05:59:25.50027",{"slug":5322,"name":5322,"fn":5323,"description":5324,"org":5325,"tags":5326,"stars":26,"repoUrl":27,"updatedAt":5336},"new-app","build new applications from scratch","Workflow for creating new applications from scratch. Covers requirements gathering, tech stack selection, scaffolding, implementation, and delivery of a functional prototype.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5327,5330,5333],{"name":5328,"slug":5329,"type":16},"Engineering","engineering",{"name":5331,"slug":5332,"type":16},"Prototyping","prototyping",{"name":5334,"slug":5335,"type":16},"Web Development","web-development","2026-07-16T05:59:25.157573",{"slug":5338,"name":5338,"fn":5339,"description":5340,"org":5341,"tags":5342,"stars":26,"repoUrl":27,"updatedAt":5350},"qc-helper","provide Qwen Code usage and configuration support","Answer any question about Qwen Code usage, features, configuration, and troubleshooting by referencing the official user documentation. Also helps users view or modify their settings.json. Invoke with `\u002Fqc-helper` followed by a question, e.g. `\u002Fqc-helper how do I configure MCP servers?` or `\u002Fqc-helper change approval mode to yolo`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5343,5346,5347],{"name":5344,"slug":5345,"type":16},"Configuration","configuration",{"name":5301,"slug":5302,"type":16},{"name":5348,"slug":5349,"type":16},"Reference","reference","2026-07-16T05:59:29.118413",{"slug":5352,"name":5352,"fn":5353,"description":5354,"org":5355,"tags":5356,"stars":26,"repoUrl":27,"updatedAt":5363},"review","review code for quality and security","Review changed code for correctness, security, code quality, and performance. Use when the user asks to review code changes, a PR, or specific files. Invoke with `\u002Freview`, `\u002Freview \u003Cpr-number>`, `\u002Freview \u003Cfile-path>`, or `\u002Freview \u003Cpr-number> --comment` to post inline comments on the PR. Add `--effort low|medium|high` to trade depth for speed (defaults to high for PRs, medium for local changes).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5357,5360],{"name":5358,"slug":5359,"type":16},"Code Review","code-review",{"name":5361,"slug":5362,"type":16},"Security","security","2026-07-30T05:30:17.682893",{"slug":5365,"name":5365,"fn":5366,"description":5367,"org":5368,"tags":5369,"stars":26,"repoUrl":27,"updatedAt":5377},"simplify","clean up and simplify code changes","Review recent code changes for reuse, code quality, and efficiency, then directly apply straightforward cleanup improvements. Use when the user wants a post-implementation cleanup pass, pre-PR polish, or asks to simplify\u002Frefine recent changes. Invoke with `\u002Fsimplify` or `\u002Fsimplify \u003Cfocus>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5370,5373,5374],{"name":5371,"slug":5372,"type":16},"Code Analysis","code-analysis",{"name":5328,"slug":5329,"type":16},{"name":5375,"slug":5376,"type":16},"Performance","performance","2026-07-20T05:58:46.653811",{"slug":5379,"name":5379,"fn":5380,"description":5381,"org":5382,"tags":5383,"stars":26,"repoUrl":27,"updatedAt":5389},"stuck","diagnose and debug stuck Qwen Code sessions","Diagnose frozen, stuck, or slow Qwen Code sessions on this machine. Scans for problematic processes, high CPU\u002Fmemory usage, hung subprocesses, and debug logs. Use \u002Fstuck or \u002Fstuck \u003CPID> to focus on a specific process.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5384,5387,5388],{"name":5385,"slug":5386,"type":16},"Debugging","debugging",{"name":5328,"slug":5329,"type":16},{"name":5375,"slug":5376,"type":16},"2026-07-16T05:59:25.838629",{"slug":5391,"name":5391,"fn":5392,"description":5393,"org":5394,"tags":5395,"stars":5402,"repoUrl":5403,"updatedAt":5404},"open-computer-use","configure Open Computer Use MCP server","Platform-neutral guidance for using Open Computer Use, the open-source Computer Use MCP server and CLI for macOS, Linux, and Windows. Use when an agent needs to install, verify, troubleshoot, configure, or operate Open Computer Use through its native CLI, stdio MCP server, or direct Computer Use tool calls.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5396,5397,5400,5401],{"name":5262,"slug":5263,"type":16},{"name":5398,"slug":5399,"type":16},"CLI","cli",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},176,"https:\u002F\u002Fgithub.com\u002FQwenLM\u002Fopen-computer-use","2026-07-16T05:59:22.010839",{"slug":5406,"name":5406,"fn":5407,"description":5408,"org":5409,"tags":5410,"stars":5420,"repoUrl":5421,"updatedAt":5422},"auto-pr","automate pull request submission and review","Automated PR submission assistant, including code review, documentation generation, and PR creation",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5411,5412,5413,5414,5417],{"name":5262,"slug":5263,"type":16},{"name":5358,"slug":5359,"type":16},{"name":5301,"slug":5302,"type":16},{"name":5415,"slug":5416,"type":16},"GitHub","github",{"name":5418,"slug":5419,"type":16},"Pull Requests","pull-requests",136,"https:\u002F\u002Fgithub.com\u002FQwenLM\u002Fqwen-code-examples","2026-07-16T06:00:01.890524",21,{"items":5425,"total":5468},[5426,5431,5438,5444,5450,5456,5462],{"slug":5256,"name":5256,"fn":5257,"description":5258,"org":5427,"tags":5428,"stars":26,"repoUrl":27,"updatedAt":5267},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5429,5430],{"name":5262,"slug":5263,"type":16},{"name":5265,"slug":5266,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":5432,"tags":5433,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5434,5435,5436,5437],{"name":14,"slug":15,"type":16},{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":5276,"name":5276,"fn":5277,"description":5278,"org":5439,"tags":5440,"stars":26,"repoUrl":27,"updatedAt":5290},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5441,5442,5443],{"name":5282,"slug":5283,"type":16},{"name":5285,"slug":5286,"type":16},{"name":5288,"slug":5289,"type":16},{"slug":5292,"name":5292,"fn":5293,"description":5294,"org":5445,"tags":5446,"stars":26,"repoUrl":27,"updatedAt":5306},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5447,5448,5449],{"name":5298,"slug":5299,"type":16},{"name":5301,"slug":5302,"type":16},{"name":5304,"slug":5305,"type":16},{"slug":5308,"name":5308,"fn":5309,"description":5310,"org":5451,"tags":5452,"stars":26,"repoUrl":27,"updatedAt":5320},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5453,5454,5455],{"name":5262,"slug":5263,"type":16},{"name":5315,"slug":5316,"type":16},{"name":5318,"slug":5319,"type":16},{"slug":5322,"name":5322,"fn":5323,"description":5324,"org":5457,"tags":5458,"stars":26,"repoUrl":27,"updatedAt":5336},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5459,5460,5461],{"name":5328,"slug":5329,"type":16},{"name":5331,"slug":5332,"type":16},{"name":5334,"slug":5335,"type":16},{"slug":5338,"name":5338,"fn":5339,"description":5340,"org":5463,"tags":5464,"stars":26,"repoUrl":27,"updatedAt":5350},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5465,5466,5467],{"name":5344,"slug":5345,"type":16},{"name":5301,"slug":5302,"type":16},{"name":5348,"slug":5349,"type":16},10]