Tools
Every tool the MCP server offers, its inputs and what it returns, read from the server itself.
The server offers 17 tools. This page is built from the server's own tool list, the one your agent app reads when it connects, so it always matches what runs.
| Tool | Permission | Read-only |
|---|---|---|
screenshot | read | yes |
read_tree | read | yes |
zoom | read | yes |
act | control | no |
wait | read | yes |
exec | exec | no |
exec_status | exec | yes |
list_screens | read | yes |
status | read | yes |
viewer_link | read | no |
clipboard | control | no |
read_file | exec | yes |
write_file | exec | no |
apps | read | yes |
page | control | no |
run_script | control | no |
guide | read | yes |
Every tool takes screen, a screen id or name, except list_screens. Leave it out when the agent can use only one screen.
screenshot
The screen now, as an image, with its size and screenshot_id. windows: true lists the open windows (id, title, app, place, state). marks: true draws each element id of the active window on the image, for element in act.
Needs read. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
windows | boolean | ||
marks | boolean |
read_tree
The accessibility tree of the active window (or window, or the part in region) as text rows: [e12] role "name" @x,y wxh (states). Act on the ids with act (element, set_value, invoke) and the same screenshot_id; any input makes them stale.
Needs read. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
window | string (like win_[0-9a-f]{1,8}) or "all" | A window id, or "all" for every window (use with query to find something). Default: active. | |
region | 4 of integer 0 to 9999 | x0,y0,x1,y1 in the screenshot: keep the elements there. | |
query | string (at most 200) | Keep rows whose role, name or value contains this (a/b), with their parents. | |
max_elements | integer 1 to 1000 | ||
screenshot_id | string (at most 40) | Bind the ids to this screenshot and use its pixels. |
zoom
A region of a screenshot at full resolution, for small text. The region is x0,y0,x1,y1 in that screenshot.
Needs read. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
screenshot_id | string (at most 40) | yes | |
region | 4 of integer 0 to 9999 | yes | |
max_upscale | "fit" or number 1 to 8 |
act
1 to 20 actions in order (click, move, drag, scroll, type, key, set_value, invoke, select, check, launch, window, focus and the rest), planned on screenshot_id, by point or by element id from read_tree, then the screen after them, taken once it stops changing (up to 2 s), so short waits are rarely needed. To wait for an app or dialog to open, use expect with its window_title (up to 10 s) rather than a long wait. tree adds the tree after. No screenshot or zoom inside.
Needs control. Can change what is on the screen.
| Input | Type | Required | Notes |
|---|---|---|---|
screenshot_id | string (at most 40) | yes | |
actions | list of object, 1 to 20 | yes | |
space | 2 of integer 0 to 9999 | ||
screenshot | boolean | false: no image back, only the result as text with the screenshot_id after it, for the next act. Default true. | |
tree | boolean or object | Also return the tree after the actions, with fresh ids: the window acted on by element, else the active one. true, or { window, query, max_elements }. | |
expect | object | Not an action: after them, wait until all hold or timeout_ms; reports satisfied, unsatisfied or unknown. { window_title: "Calc", timeout_ms: 8000 }. | |
expect.conditions | list of object, 1 to 8 | All must hold. | |
expect.conditions[].element | string (like e[0-9]{1,5}) | ||
expect.conditions[].window_title | string (at most 200) | The active window's title contains this. | |
expect.conditions[].value | string (at most 503) | ||
expect.conditions[].value_contains | string (at most 503) | ||
expect.conditions[].name_contains | string (at most 203) | ||
expect.conditions[].enabled | boolean | ||
expect.conditions[].selected | boolean | ||
expect.conditions[].checked | boolean | ||
expect.conditions[].focused | boolean | ||
expect.conditions[].text | string (at most 200) | Text anywhere in the active window's tree. | |
expect.conditions[].gone | boolean | With text: wait until it is absent. | |
expect.window_title | string (at most 200) | Short for conditions: [{ window_title }]. | |
expect.text | string (at most 200) | Short for conditions: [{ text }]. | |
expect.gone | boolean | ||
expect.timeout_ms | integer 0 to 10000 | How long to wait for them; default 2000. |
wait
Pause up to 20 seconds; returns an image only if the screen changed since screenshot_id.
Needs read. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
screenshot_id | string (at most 40) | yes | |
seconds | number 0 to 20 | yes |
exec
Run a shell command as the desktop user. Waits up to 25 s: then the exit code and the last 8 KiB of output, else an exec id for exec_status.
Needs exec. Can change what is on the screen.
| Input | Type | Required | Notes |
|---|---|---|---|
command | string (at most 16384) | yes | |
cwd | string (at most 4096) | ||
timeout_s | integer 0 to 86400 |
exec_status
Wait up to 25 s for a command started by exec to finish, and read its output from the offsets given.
Needs exec. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
exec_id | string (at most 40) | yes | |
stdout_from | integer 0 to 9007199254740991 | ||
stderr_from | integer 0 to 9007199254740991 |
list_screens
The screens you can use: id, name, kind and state.
Needs read. Read-only.
No inputs.
status
A screen's state and health.
Needs read. Read-only.
No inputs.
viewer_link
A view-only live link to show the person what is happening. Open within 60 s, once; it ends after 60 minutes.
Needs read. Not read-only.
No inputs.
clipboard
Read the screen's clipboard text, or set it when text is given. To paste text into the focused field, use act's type with via: paste.
Needs control. Not read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
text | string (at most 100000) |
read_file
Read a file on the screen, up to 64 KiB: text as text, anything else as base64. Bigger files: the API, the SDK, or exec.
Needs exec. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
path | string (at most 1024) | yes | Absolute, or relative to /home/user. |
write_file
Write a file on the screen, up to 64 KiB, from text or from base64 (one of them), creating its folders and replacing what was there.
Needs exec. Can change what is on the screen.
| Input | Type | Required | Notes |
|---|---|---|---|
path | string (at most 1024) | yes | Absolute, or relative to /home/user. |
text | string (at most 65536) | ||
base64 | string (at most 87384, like [A-Za-z0-9+/]*={0,2}) |
apps
The apps installed on the screen: id, name, what each is for, whether it is open. Start one with act [{ type: "launch", app }].
Needs read. Read-only.
No inputs.
page
A browser tab by ref: op read lists links, buttons and fields as [r3] textbox "Email" = ""; click, fill (value), select (value), check (value "true"/"false"), press (keys), navigate (url), back act on a ref and answer with the page after. prepare says whether the browser can be read.
Needs control. Can change what is on the screen.
| Input | Type | Required | Notes |
|---|---|---|---|
op | one of prepare, read, click, fill, select, check, press, navigate, back | yes | |
ref | string (like r[0-9]{1,5}) | ||
value | string (at most 10000) | ||
keys | string (like [A-Za-z0-9_+ -]{1,100}) | A key or +-joined chord: enter, ctrl+s, alt+tab. xdotool, OpenAI and Playwright names work. | |
url | string (at most 2000) | ||
query | string (at most 200) | read: only rows containing this (a/b). | |
tab | string (at most 64) | ||
relaunch | boolean | prepare: restart the browser with its debugging port. |
run_script
Run JavaScript inside the screen that calls screenshot(), tree(q), find(q), act(actions, opts), wait(ms), expect(cond), apps(), launch(id), page(req), log(); for loops and read-then-act in one call. return gives the result. 28 s at most. screenshot: true attaches the screen after. guide topic scripts has examples.
Needs control. Can change what is on the screen.
| Input | Type | Required | Notes |
|---|---|---|---|
code | string (at most 32768) | yes | |
screenshot | boolean |
guide
How to use these tools well, by topic: loop, elements, apps (and windows), browser, scripts, errors.
Needs read. Read-only.
| Input | Type | Required | Notes |
|---|---|---|---|
topic | one of loop, elements, apps, browser, scripts, errors | yes |