Actions
Click, type, press keys, scroll, drag and move the pointer. One action per call, or a batch of up to 20 in one round trip.
Run a batch of actions
Needs the control permission. Answers within 28 s.
Runs 1 to 20 input actions in order, 100 ms apart: click, move, drag, scroll, type, key, wait, settle, zoom and more. Stops at the first failure (status partial, later actions not_executed). With screenshot: after the reply carries the screen after the last action that ran, once it settles, so one call is one agent step.
Coordinates are in space: by default the size of the screenshot named by screenshot_id, else 1280x720. Points outside are rejected, never clamped. With screenshot_id, nothing runs if any input ran after that shot (a person's in the live view included) or the screen restarted (409 STALE_OBSERVATION).
There is no lock between credentials: any credential with control may act at any time. Input on one screen runs one call at a time, in the order calls arrive, while commands run side by side. While a person has control of the screen, calls get 409 CONTROL_HELD; a person taking control mid-batch stops it at the next action (status partial, later actions not_executed). The optional Kubi-Run header groups calls into a run in the screen's history.
Send an Idempotency-Key: after a lost reply, resend once with the same key. Within 10 minutes you get the stored per-action results and a fresh screenshot (Idempotent-Replayed: true), never a second click. If the screen restarted in between, 504 OUTCOME_UNKNOWN: look, never repeat blindly.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Run a batch of actions › Request Body
Run in order, 100 ms apart. Stops at the first failure; the rest are not_executed. The whole batch must fit in 28 s.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
Run a batch of actions › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Click
Needs the control permission. Answers within 10 s.
Clicks at x, y, or at the pointer when both are left out. button picks left, right or middle; count: 2 is a double click and count: 3 a triple click. modifiers holds keys down while clicking.
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Click › Request Body
xPixels in the request's coordinate space, origin top left.
yPixels in the request's coordinate space, origin top left.
buttoncountmodifiersKeys held during the action.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Click › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Type text
Needs the control permission. Answers within 28 s.
Types text as keystrokes into whatever has focus, up to 1,000 characters. enter: true presses Enter afterwards. The result reports how many characters were delivered.
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Type text › Request Body
textenterdelay_msviakeys: keystrokes, up to 1,000 characters. paste: put the text on the clipboard and paste it (ctrl+v, or ctrl+shift+v in a terminal), up to 100,000 characters, for long text and characters the keyboard lacks. It leaves the text on the clipboard.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Type text › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Press keys
Needs the control permission. Answers within 25 s.
Presses one key or a chord: enter, ctrl+s, alt+tab. repeat presses it several times; hold_ms keeps it down (up to 20 s).
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Press keys › Request Body
keys^[A-Za-z0-9_+ -]{1,1… · requiredA key or +-joined chord: enter, ctrl+s, alt+tab. xdotool names (Return), OpenAI's upper-case names and Playwright names are accepted as aliases.
repeathold_msscreenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Press keys › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Scroll
Needs the control permission. Answers within 10 s.
Scrolls at x, y, or at the pointer. Positive dy scrolls down and positive dx right, in wheel notches (or pixels with unit: px, 100 px per notch).
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Scroll › Request Body
xPixels in the request's coordinate space, origin top left.
yPixels in the request's coordinate space, origin top left.
dxdyunitmodifiersKeys held during the action.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Scroll › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Drag
Needs the control permission. Answers within 10 s.
Presses at the first point of path, moves through the rest and releases at the last: for sliders, selections and moving files or windows.
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Drag › Request Body
pathbuttonmodifiersKeys held during the action.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Drag › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.
Move the pointer
Needs the control permission. Answers within 10 s.
Moves the pointer to x, y without clicking, to open hover menus and tooltips.
Works like a batch of one: screenshot: after returns the screen once it settles, screenshot_id refuses to act on a stale screen (409 STALE_OBSERVATION), and resending with the same Idempotency-Key returns the stored result instead of acting twice. For several steps in one round trip, use runActions.
path Parameters
screen_id^scr_[0-9a-z]{20}$ · requiredScreen id (scr_...).
Headers
Idempotency-Key^[ -~]+$Any unique string, ideally a UUIDv7; the SDK and MCP always send one. A retry with the same key and body gets the stored answer instead of running twice (kept 24 h for screens and power, 10 minutes for actions and commands). Stored once execution began: 2xx, 500 and 504. Never stored when outcome is not_applied (400, 401, 403, 404, 409, 415, 422, 429), so the same key can be retried after a conflict or a rate limit. Replays of runActions keep per-action results and shot metadata only, capture a fresh screenshot and carry Idempotent-Replayed: true. Same key, different body: 422 IDEMPOTENCY_KEY_REUSED. Same key still running: 409 IDEMPOTENCY_KEY_IN_USE.
Kubi-Run^[ -~]{1,100}$Optional label that groups calls into one run in the screen's history. It is only a label and never locks the screen. The SDK fills it from KUBI_RUN, else Claude Code's CLAUDE_CODE_SESSION_ID, else one id per client it creates.
Move the pointer › Request Body
xPixels in the request's coordinate space, origin top left.
yPixels in the request's coordinate space, origin top left.
screenshot_id^shot_[0-9a-z]{20}$Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.
screenshotafter: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.
How the after-shot is taken.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
Move the pointer › Responses
Success.
partial: an action failed; see results. The after-screenshot is still attached.
space[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.
screenshot_unchangedif_changed found no visible change.
The window with the keyboard focus after the batch. Left out when there is none.
Windows that opened during the batch, newest first. Left out when none did.