The look and act loop
Screenshots, actions, coordinates, and how one call becomes one agent step.
An agent on a screen works in a loop: look at the screen, decide, act, look again. The API is shaped so each turn of that loop is one call.
Look
GET /screens/{id}/screenshot returns JSON with a base64 image (1280x720 PNG by default) and a screenshot_id. For plain image bytes, for curl -o or an <img> tag, use /screenshot/image.
To read small text, zoom: pass region as the top-left and bottom-right corners (x0,y0,x1,y1) and get that part at full resolution.
Add windows=true to also get the open windows, topmost first: each one's title, app, place in the screenshot's pixels, and which one is active.
Act
Every common action has its own endpoint:
| Endpoint | Does |
|---|---|
POST /screens/{id}/click | Left, right or middle click; count: 2 double-clicks |
POST /screens/{id}/type | Types text, optionally presses Enter; via: "paste" pastes it |
POST /screens/{id}/press | A key or chord: enter, ctrl+s, alt+tab |
POST /screens/{id}/scroll | Wheel notches at a point |
POST /screens/{id}/drag | Press, move through points, release |
POST /screens/{id}/move | Move the pointer to hover |
POST /screens/{id}/actions | Up to 20 of any of these in one call |
Paste long text
type sends keystrokes, up to 1,000 characters. For longer text, or characters the keyboard lacks, send "via": "paste": the text goes on the screen's clipboard and is pasted into the focused field (ctrl+v, or ctrl+shift+v in a terminal), up to 100,000 characters. The text stays on the clipboard afterwards.
GET /screens/{id}/clipboard reads the clipboard's text, for something the agent copied in an app; PUT sets it without pasting. Both need control, since clipboards hold copied passwords. History keeps the text an agent sets, as it keeps typed text, and never the text it reads.
One call, one step
Add "screenshot": "after" to any action and the reply carries the screen once it settles. The agent sees the result of its click without a second request.
Code
What an action did
Each click, key, typing or other input in the reply says what happened, without a second look:
screen_changed: whether anything visible changed after it (a moving pointer or a blinking caret doesn't count).falsemeans nothing changed on screen, not that it failed: a slow app may still be working, so look before you repeat it.window: the window it reached, by title and app. For the pointer, the topmost window under the point; for keys and typing, the active window.
The reply also gives active_window, the window with the keyboard focus after the batch, and windows_opened when a dialog or app opened during it.
Never click a stale screen
Pass the screenshot_id you decided on. If anything changed since (another input ran, a person's in the live view included, or the screen restarted), nothing runs and you get 409 STALE_OBSERVATION. Take a new screenshot and decide again.
Coordinates
Coordinates are pixels of the screenshot you acted on. Models that think in another grid can say so: Gemini answers on a 0 to 999 grid, so send "space": [1000, 1000] and we convert. Points outside the space are rejected, never clamped.
Batches
POST /screens/{id}/actions runs up to 20 actions in order, 100 ms apart, and stops at the first failure. Use it when the agent already knows several steps: click a search box, type, press Enter. Add a settle action to wait for a menu to open before the next click. The MCP act tool always uses batches.
When a person steps in
A person can take control in the live view at any time. Until they finish, calls that look or act get 409 CONTROL_HELD, with the seconds left in retry_after. Nothing ran. Wait, then take a new screenshot: their input has made your last one stale, and acting on it gets 409 STALE_OBSERVATION. See People and agents.