# The look and act loop

> Screenshots, actions, coordinates, and how one call becomes one agent step.

An agent on a screen works in a loop: **look** at the screen, decide, **act**, look again. The API is shaped so each turn of that loop is one call.

## Look

`GET /screens/{id}/screenshot` returns JSON with a base64 image (1280x720 PNG by default) and a `screenshot_id`. For plain image bytes, for `curl -o` or an `<img>` tag, use `/screenshot/image`.

To read small text, zoom: pass `region` as the top-left and bottom-right corners (`x0,y0,x1,y1`) and get that part at full resolution.

Add `windows=true` to also get the open windows, topmost first: each one's title, app, place in the screenshot's pixels, and which one is active.

## Act

Every common action has its own endpoint:

| Endpoint | Does |
|---|---|
| `POST /screens/{id}/click` | Left, right or middle click; `count: 2` double-clicks |
| `POST /screens/{id}/type` | Types text, optionally presses Enter; `via: "paste"` pastes it |
| `POST /screens/{id}/press` | A key or chord: `enter`, `ctrl+s`, `alt+tab` |
| `POST /screens/{id}/scroll` | Wheel notches at a point |
| `POST /screens/{id}/drag` | Press, move through points, release |
| `POST /screens/{id}/move` | Move the pointer to hover |
| `POST /screens/{id}/actions` | Up to 20 of any of these in one call |

## Paste long text

`type` sends keystrokes, up to 1,000 characters. For longer text, or characters the keyboard lacks, send `"via": "paste"`: the text goes on the screen's clipboard and is pasted into the focused field (ctrl+v, or ctrl+shift+v in a terminal), up to 100,000 characters. The text stays on the clipboard afterwards.

`GET /screens/{id}/clipboard` reads the clipboard's text, for something the agent copied in an app; `PUT` sets it without pasting. Both need `control`, since clipboards hold copied passwords. History keeps the text an agent sets, as it keeps typed text, and never the text it reads.

## One call, one step

Add `"screenshot": "after"` to any action and the reply carries the screen once it settles. The agent sees the result of its click without a second request.

```json
{"x": 450, "y": 280, "screenshot_id": "shot_01jb7q2x9k4m8q1w3z5t", "screenshot": "after"}
```

## What an action did

Each click, key, typing or other input in the reply says what happened, without a second look:

- `screen_changed`: whether anything visible changed after it (a moving pointer or a blinking caret doesn't count). `false` means nothing changed on screen, not that it failed: a slow app may still be working, so look before you repeat it.
- `window`: the window it reached, by title and app. For the pointer, the topmost window under the point; for keys and typing, the active window.

The reply also gives `active_window`, the window with the keyboard focus after the batch, and `windows_opened` when a dialog or app opened during it.

## Never click a stale screen

Pass the `screenshot_id` you decided on. If anything changed since (another input ran, a person's in the live view included, or the screen restarted), nothing runs and you get `409 STALE_OBSERVATION`. Take a new screenshot and decide again.

## Coordinates

Coordinates are pixels of the screenshot you acted on. Models that think in another grid can say so: Gemini answers on a 0 to 999 grid, so send `"space": [1000, 1000]` and we convert. Points outside the space are rejected, never clamped.

## Batches

`POST /screens/{id}/actions` runs up to 20 actions in order, 100 ms apart, and stops at the first failure. Use it when the agent already knows several steps: click a search box, type, press Enter. Add a `settle` action to wait for a menu to open before the next click. The MCP `act` tool always uses batches.

## When a person steps in

A person can take control in the live view at any time. Until they finish, calls that look or act get `409 CONTROL_HELD`, with the seconds left in `retry_after`. Nothing ran. Wait, then take a new screenshot: their input has made your last one stale, and acting on it gets `409 STALE_OBSERVATION`. See [People and agents](/concepts/people-and-agents).