Guides
Claude, OpenAI and Gemini
How each model's computer-use actions map to ours, and which coordinates to send.
Computer-use models describe actions in their own format. Ours covers all three, so an adapter only renames fields. The SDK ships one for each model.
Action mapping
Ours (type) | Claude | OpenAI | Gemini |
|---|---|---|---|
click (button, count 1 to 3) | left, right, middle, double, triple click | click, double_click | click, double, triple, middle, right click |
move | mouse_move | move | move |
drag (path) | left_click_drag | drag (path) | drag_and_drop |
scroll (dx, dy) | scroll (direction, amount) | scroll (scroll_x, scroll_y in px) | scroll (direction, magnitude in px) |
type (text, enter) | type | type | type (press_enter) |
key (keys, repeat, hold_ms) | key, hold_key | keypress | press_key, hotkey |
mouse_down, mouse_up | left_mouse_down, left_mouse_up | none | mouse_down, mouse_up |
key_down, key_up | none | none | key_down, key_up |
wait (ms) | wait (seconds) | wait | wait (seconds) |
zoom (region) | zoom | none | none |
screenshot | screenshot | screenshot | take_screenshot |
Coordinates
- Claude and OpenAI answer in pixels of the screenshot you sent. Pass its
screenshot_idand you are done. - Gemini answers on a 0 to 999 grid. Send
"space": [1000, 1000]with the action and we convert.
Screenshot size
Shots come back at 1280x720 by default. That costs about 1,200 image tokens per shot on Claude, OpenAI and Gemini, against about 2,700 at 1920x1080, and small text stays readable. Zoom into a region when text is too small.
Long waits
Claude's wait and hold_key can ask for up to 300 seconds. Our calls answer within 30, so the SDK adapter splits long waits into several calls.
Using an agent app instead of your own loop? Connect your agent gives it all of this through MCP with no adapter.