Schemas

INVALID_ARGUMENT: the field at fault, as a path (actions[0].y).

INVALID_ARGUMENT, URL_NOT_ALLOWED: what is wrong.

UNAUTHENTICATED: why the credential failed.

read: see screens, screenshots and history. control: act on the GUI. exec: run commands, read, write and list files. power: create, start, stop, restart (and, for API keys, delete). Same names as OAuth scopes.

STALE_OBSERVATION: input actions run since your screenshot.

STALE_OBSERVATION: the screen restarted since your screenshot.

QUOTA_EXCEEDED: screens the organization has, running or stopped.

QUOTA_EXCEEDED: the most screens the organization may have.

QUOTA_EXCEEDED on createSnapshot: the screen's snapshots.

QUOTA_EXCEEDED on createSnapshot: the most a screen may keep.

INVALID_ARGUMENT on a top-up: the smallest allowed now.

INVALID_ARGUMENT on a top-up: the largest allowed now.

Stable machine code. The list can grow: treat an unknown code by its HTTP status.

• INVALID_ARGUMENT (400): A field is missing, malformed or out of range (details.field, details.problem). Fix that field and resend.
• COORDINATE_OUT_OF_RANGE (400): A point lies outside the coordinate space; it is rejected, never clamped. Use a point inside details.space.
• BATCH_TOO_LONG (400): The batch would run past 28 s, inside the 30 s call limit. Split it into smaller batches.
• UNAUTHENTICATED (401): The credential is missing, malformed, unknown, revoked or expired (details.reason). Send a valid key or token.
• QUOTA_EXCEEDED (402): The organization is at its screen cap (details.screen_cap). Delete a screen, or a person asks support for a higher cap.
• PAYMENT_REQUIRED (402): Payment failed. A person pays at details.pay_url.
• CREDIT_EXHAUSTED (402): The usage credit that pays for disposable screens is used up (details.balance_usd). A person tops up at details.billing_url, or create a persistent screen.
• SUBSCRIPTION_REQUIRED (402): Buying usage credit needs a paid persistent screen first. A person subscribes at details.billing_url.
• PERMISSION_DENIED (403): The credential lacks details.needed, or is not bound to this screen. Use a credential that has it.
• HUMAN_ONLY (403): Only a signed-in person (dashboard cookie) may call this. Ask a person to do it in the dashboard.
• CSRF_REJECTED (403): A cookie request changed state without the dashboard's Origin. Send it from the dashboard, or use an API key.
• ORGANIZATION_SUSPENDED (403): The organization is suspended. Stop and tell the user.
• ACCOUNT_PENDING (403): The account is waiting for approval. Stop and tell the user.
• SCREEN_NOT_FOUND (404): No such screen in your organization (the same answer for another organization's id). List screens for a valid id.
• NOT_FOUND (404): No such operation, exec, key, app, export, template, snapshot or saved screenshot (saved ones go after 30 days). Check the id.
• FILE_NOT_FOUND (404): Nothing at that path on the screen. List the folder with listFiles.
• NO_CAPACITY (409): No free machine for a new screen right now; nothing was charged. Retry after retry_after.
• STALE_OBSERVATION (409): Input ran (anyone's, a person's in the live view included) or the screen restarted since your screenshot (details.acted_since, details.restarted). Take a new screenshot first.
• CONTROL_HELD (409): A person has control of this screen, so nothing ran. Control ends when they release it, or after 5 minutes with no input from them. Wait retry_after seconds, then take a new screenshot before acting.
• SCREEN_NOT_RUNNING (409): The screen is stopped, starting or stopping. Start it, or wait on its operation.
• OPERATION_IN_PROGRESS (409): Another power operation runs on this screen. Wait on it, then retry.
• IDEMPOTENCY_KEY_IN_USE (409): A request with this key is still running. Retry the same request.
• UNSUPPORTED_CAPABILITY (409): This screen cannot do that (for example a screenless box). Use a screen that can.
• PAYLOAD_TOO_LARGE (413): The body or file is over details.limit_bytes. Send less, or move big files through exec.
• UNSUPPORTED_MEDIA_TYPE (415): Wrong Content-Type. Send application/json (application/octet-stream for file uploads).
• IDEMPOTENCY_KEY_REUSED (422): This key was used with a different body. Use a new key.
• SECRET_BLOCKED (422): The input contains one of our API keys; nothing was typed or run. Remove the key and resend.
• URL_NOT_ALLOWED (422): The URL is not public HTTPS: it resolves to a private or reserved address, or redirects. Use a public HTTPS URL.
• RATE_LIMITED (429): Too many requests. Wait retry_after seconds.
• INTERNAL (500): Our bug. Retry with the same Idempotency-Key.
• SCREEN_UNAVAILABLE (503): The screen's host or desktop is not answering. Retry after retry_after.
• OUTCOME_UNKNOWN (504): The work may or may not have happened. Look before acting; never resend blindly.

One sentence for people and agents, ending with the fix. Do not parse it.

Quote it in support requests.

True when the same request may succeed later. The SDK retries only these, never 504.

not_applied: nothing changed, so the reply is never stored for the Idempotency-Key. unknown: look before acting.

Code-specific facts. Every key is optional and new keys may appear (U6).

The same shape for every error on every route.

[x, y] in the coordinate space.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

[x0, y0, x1, y1], top-left and bottom-right corners, in the coordinate space.

Keys held during the action.

A person has control. Calls that look at or act on the screen get 409 CONTROL_HELD.

Seconds it has run as a disposable screen.

Credit charged for it so far, the step now running included.

When it became disposable (created or switched).

persistent: on the monthly subscription, one price per screen, running or stopped; kept until deleted, and a delete can be undone for 7 days. disposable: paid from usage credit for the time it runs, charged per billing step as each step starts; it can carry an end time, is deleted 7 days after it stops unless started again, and a delete is final.

The cube's colour (ScreenColor). New colours may appear: show an unknown one as green.

Your own key-value tags. Filter lists with label=key:value.

Power state. failed carries a state_reason. New values may appear.

The image it was created from. One desktop image at launch.

The size it was sold as (standard, the one size at launch); resources says what it has.

One size at launch. Resize comes later.

Each capability on its own: a running VM can have a broken display. Host unreachable means unknown.

Whether a person has control of the screen.

Goes up when the machine restarts. Old screenshots and exec ids stop applying.

Live views open on the screen right now, at most 3.

Unique in the organization. Default: generated.

The colour of the screen's cube, shown beside its name in the dashboard. Only for people telling screens apart; nothing else reads it. A new screen is green unless its create says otherwise.

Your own key-value tags. Filter lists with label=key:value.

persistent: on the monthly subscription, one price per screen, running or stopped; kept until deleted, and a delete can be undone for 7 days. disposable: paid from usage credit for the time it runs, charged per billing step as each step starts; it can carry an end time, is deleted 7 days after it stops unless started again, and a delete is final.

Start from this template. Default: blank.

Default: standard, the one size at launch. A size not sold in the screen's kind: INVALID_ARGUMENT.

Disposable only: delete it this many seconds after it is created. Default: no end time.

The colour of the screen's cube, shown beside its name in the dashboard. Only for people telling screens apart; nothing else reads it. A new screen is green unless its create says otherwise.

Your own key-value tags. Filter lists with label=key:value.

persistent: on the monthly subscription, one price per screen, running or stopped; kept until deleted, and a delete can be undone for 7 days. disposable: paid from usage credit for the time it runs, charged per billing step as each step starts; it can carry an end time, is deleted 7 days after it stops unless started again, and a delete is final.

saving: being made from its screen.

One size at launch. Resize comes later.

true: a screen started from it comes back with the open windows it was saved with. false: it boots from the saved disk.

Screens started from it so far.

Unique among the organization's templates.

creating: being taken.

Optional label.

reconciling: the outcome is uncertain and we are checking the host. Not a failure; do not retry.

Names one capture. Pass it to runActions (and to getScreenshot with region) so coordinates use its size and stale views are refused.

Input actions run on this screen before the capture. runActions with this screenshot_id refuses to run if it has moved on.

Encoded image size before base64.

The image, base64. A copy is saved in the screen's history unless detailed history is off.

With windows=true: the open windows, topmost first, in this screenshot's pixels (a region's: the full screenshot's). Normal windows and dialogs only; at most 50.

The window's title, clipped to 200 characters. Read off the screen: history keeps the screenshot, never this text.

The app, from the window's class (for example chromium, org.gnome.TextEditor).

The window's title, clipped to 200 characters. Read off the screen: history keeps the screenshot, never this text.

The app, from the window's class (for example chromium, org.gnome.TextEditor).

true for the window with the keyboard focus.

WIDTHxHEIGHT with the desktop's aspect ratio, or full (desktop size).

png by default (lossless, sharp text, no ICC chunk). webp at quality 80 is what the hosted MCP server sends, because it is smaller. jpeg is not recommended for UI text.

webp and jpeg only.

Cap on the encoded image. webp and jpeg step quality 80, 70, 60, 50, then shrink; png only shrinks. The reply says the quality and size used. The hosted MCP sends 105000.

Draw the pointer.

Wait until nothing changed for this long (pointer and caret-sized changes ignored).

Then capture anyway.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Keys held during the action.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Keys held during the action.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Keys held during the action.

keys: keystrokes, up to 1,000 characters. paste: put the text on the clipboard and paste it (ctrl+v, or ctrl+shift+v in a terminal), up to 100,000 characters, for long text and characters the keyboard lacks. It leaves the text on the clipboard.

A key or +-joined chord: enter, ctrl+s, alt+tab. xdotool names (Return), OpenAI's upper-case names and Playwright names are accepted as aliases.

A key or +-joined chord: enter, ctrl+s, alt+tab. xdotool names (Return), OpenAI's upper-case names and Playwright names are accepted as aliases.

A key or +-joined chord: enter, ctrl+s, alt+tab. xdotool names (Return), OpenAI's upper-case names and Playwright names are accepted as aliases.

WIDTHxHEIGHT with the desktop's aspect ratio, or full (desktop size).

png by default (lossless, sharp text, no ICC chunk). webp at quality 80 is what the hosted MCP server sends, because it is smaller. jpeg is not recommended for UI text.

Cap on the encoded image. webp and jpeg step quality 80, 70, 60, 50, then shrink; png only shrinks. The reply says the quality and size used. The hosted MCP sends 105000.

[x0, y0, x1, y1], top-left and bottom-right corners, in the coordinate space.

png by default (lossless, sharp text, no ICC chunk). webp at quality 80 is what the hosted MCP server sends, because it is smaller. jpeg is not recommended for UI text.

Cap on the encoded image. webp and jpeg step quality 80, 70, 60, 50, then shrink; png only shrinks. The reply says the quality and size used. The hosted MCP sends 105000.

Zoom only. fit (default): scale the region to the usual shot size, as Anthropic does. A number caps the upscale (2 = at most twice the desktop pixels), which costs fewer image tokens.

Run in order, 100 ms apart. Stops at the first failure; the rest are not_executed. The whole batch must fit in 28 s.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.

after: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.

How the after-shot is taken.

The same shape for every error on every route.

[x, y] in the coordinate space.

Input actions only: whether anything visible changed after it (pointer and caret-sized changes ignored), watched until the next action, or up to 500 ms after the last one. false means no visible change, not failure: look again before repeating it.

Input actions only: the topmost window under the point for a pointer action, else the active window that got the keys.

partial: an action failed; see results. The after-screenshot is still attached.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

if_changed found no visible change.

The window with the keyboard focus after the batch. Left out when there is none.

Windows that opened during the batch, newest first. Left out when none did.

Shell command, run with bash -lc as the desktop user. Send this or argv.

Program and arguments, no shell. Send this or command.

Written to stdin, then closed.

0: no limit (dev servers, bots). At the limit we send TERM, then KILL after 10 s; status timed_out. Runs on after the call returns.

Answer inline if done within this; else 202 and an id to poll.

Bytes of output per stream in this reply.

true: the last max_bytes after the offset (errors and exit lines). false: the first.

lost: the screen restarted mid-run; the exit status is unknown. Never rerun blindly.

Output from stdout_from, up to max_bytes (head or tail). UTF-8, invalid bytes replaced.

Total bytes written so far. Use as the next stdout_from.

Older output fell out of the 1 MiB per-stream buffer.

The clipboard's text; empty when it holds none or holds no text.

true when the clipboard held more than 100,000 characters and only the first came back.

Text to put on the clipboard; empty clears it.

interactive needs a person (cookie). Agents get view links.

Open within 60 s. Single use. The ticket rides in the URL fragment.

call for a call on the screen; otherwise an event such as screen.created, screen.deleted, key.created, key.revoked, app.connected, secret.set, proxy.set, control.taken or control.released. New kinds may appear.

Who made the call: an API key, a connected app, a signed-in person, or Kubi itself.

For example click (450, 280); type 15 chars; key enter.

ok or an error code.

true when getSavedScreenshot returns the image. false when detailed history is off or the upload failed.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.

Keys held during the action.

Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.

after: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.

How the after-shot is taken.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

keys: keystrokes, up to 1,000 characters. paste: put the text on the clipboard and paste it (ctrl+v, or ctrl+shift+v in a terminal), up to 100,000 characters, for long text and characters the keyboard lacks. It leaves the text on the clipboard.

Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.

after: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.

How the after-shot is taken.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

A key or +-joined chord: enter, ctrl+s, alt+tab. xdotool names (Return), OpenAI's upper-case names and Playwright names are accepted as aliases.

Optional precondition and coordinate default: the screenshot you acted on. If any input ran after it (a person's in the live view included) or the screen restarted, nothing runs: 409 STALE_OBSERVATION. The hosted MCP always sends it.

after: attach the screen after the last action that ran, once it settles. if_changed: only if it changed visibly since screenshot_id (else screenshot_unchanged: true). none: no image. The hosted MCP sends after.

How the after-shot is taken.

[width, height] that coordinates are expressed in. Default: the size of the screenshot named by screenshot_id, else [1280, 720] (the default screenshot). Both sent and different: 400 INVALID_ARGUMENT with both sizes in details. Gemini sends [1000, 1000]. Each axis is scaled to the desktop separately.

Pixels in the request's coordinate space, origin top left.

Pixels in the request's coordinate space, origin top left.