How computer-use agents work

In a typical loop, the system receives a task and information about the environment. The model proposes a tool action, the application executes it, and the result becomes evidence for the next decision. Anthropic's computer-use API documentation describes this exchange between tool calls and tool results. The application, not a text response alone, performs the computer action.

The distinction matters when reviewing work. A plan describes intended actions. A tool response describes an attempted action or observation. A destination artifact shows what actually exists. Treat these as different kinds of evidence rather than accepting a fluent account of the task as proof that it finished.

A screenshot gives the agent the visible layout at a particular moment. Accessibility information can expose the names, roles, and values of controls. A browser can provide page content and elements. A connector can provide structured records. None of these sources automatically represents everything relevant to a task.

For example, a visible spreadsheet may show only a few rows, while a structured read can expose a range. A selected design element may be meaningful visually even when its name is generic. Choose the source that identifies the target and contains the facts the task needs; more context is not always clearer context.

Desktop control interacts with a computer interface. Browser tools may work directly with web content. Connected tools call actions a service exposes. A task can use several paths: read an email through an integration, calculate a value with code, and present the result in a document or screen-based workspace.

A useful interface should make the consequential parts understandable: what source was read, what destination will change, and what result needs review. It does not need to expose every implementation detail. The purpose is to help you correct the task and understand the outcome.

A local desktop task and a task in a separate cloud browser start from different environments. OpenAI's cloud-browser documentation says its sessions do not automatically inherit your personal tabs or sign-ins. A task that depends on a local file or unsaved app state needs an appropriate local execution path, not an assumption that every agent sees the same computer.

Before a task starts, identify the account, file, browser session, and permissions it depends on. If the environment changes during the run, the agent needs fresh evidence. An old observation should not be treated as a permanent description of what is currently on screen.

Permission to read a screen, interact with an app, or connect an account makes a class of actions possible. It does not define the exact task you wanted. Your handoff still needs a target and a clear boundary: prepare a draft, change this selected setting, or add these rows to this named sheet.

Computer-use documentation recommends human confirmation for consequential decisions. Keep the review point specific to the task. A permission prompt and an approval to send, purchase, or overwrite are different decisions; granting one should not be interpreted as granting the other.

For a text-entry task, inspect the actual field. For a spreadsheet transfer, compare source values with destination rows. For a setting change, read the resulting setting or review the visible result. A click acknowledgement is weaker evidence than the resulting state.

When a result is uncertain, refresh the evidence before retrying. The earlier action may have completed even if its response did not arrive. This is especially important for actions that create items or alter existing work. The user-facing goal is a correct, inspectable outcome, not a long sequence of apparently successful clicks.