Computer-Use and Voice Agents: How They Work

Most of what this site covers about agents assumes text in, tool calls out. Two families of agent changed the input and output channels, and each brings its own engineering. Computer-use agents perceive a screen and act with a mouse and keyboard, which lets them use software that has no API. Voice agents listen and speak in real time, which lets them take a phone call. Both are in production now; both are more fragile than tool-calling agents; and both need harness work that text agents do not.

Computer-Use Agents

How they work

A computer-use agent runs a loop like any other, with a different perception and action space. Perception is a screenshot (sometimes plus an accessibility tree, which gives structured element names and positions). Action is a small set of primitives: move the mouse to coordinates, click, type text, press a key, scroll, wait. The model looks at the screen, decides on the next primitive, the harness executes it in a controlled environment, and a new screenshot comes back.

Two variants coexist. GUI agents act on pixels and are general: they can use anything a person can see. Command-line and script agents generate shell commands or scripts and are faster and more reliable when a terminal or API exists. Production systems often combine them: use the API or CLI where one exists, fall back to the GUI where it does not.

Where they are fragile

• Grounding. Clicking the right pixel on a dense screen is hard. Small models miss; layouts shift; a popup appears.
• Latency. Screenshot, model call, action, wait, repeat. A ten-step task is many seconds; a hundred-step task is minutes.
• State. The agent's only memory of what it did is the screenshot history. Undo is often impossible.
• Security. The agent reads everything on the screen, including whatever a web page chooses to display. Prompt injection through a rendered page is the same problem as text injection, with a larger attack surface, and the agent can act on it with a real mouse.

What the harness does differently

• Runs the agent in an isolated environment: a VM or container with a fresh browser profile, no saved credentials, and a network allowlist. Never on a person's real desktop.
• Gates side effects by observation, not by tool name. There is no send_email tool to scope; there is a Send button. The harness needs rules ("pause before clicking anything labeled submit, send, pay, delete, or confirm") and often a second model watching for those.
• Bounds the loop tightly on steps and time, because loops on a GUI are expensive and visible.
• Records video or screenshot sequences as the trace, so "what did it do" can be reviewed by a person.
• Verifies by re-observation: after an action, check the screen for the expected result rather than trusting the action succeeded.

Voice Agents

How they work

Older voice systems chained three models: speech-to-text, a language model, text-to-speech. It worked and it was slow, and the seams showed: no interruptions, flat delivery, lost nuance. Newer audio-native models process and generate audio directly in one model, which gives sub-second responses, natural turn-taking, the ability to be interrupted, and expressive output. The agent loop is the same underneath: the model can call tools mid-conversation (look up an order, book a slot) and speak the result.

A production voice agent connects to telephony (SIP or a provider's phone integration) or a web audio stream, runs the model with a streaming connection, and keeps a text transcript alongside for logging, evaluation, and tool calls.

Where they are fragile

• Turn-taking. Knowing when the person is done talking, handling interruptions, not talking over them. Audio-native models are much better; still imperfect on noisy lines.
• Recognition under noise, accents, and jargon. Names, addresses, product codes. Confirm anything that will be acted on.
• Latency budgets are unforgiving. A pause longer than about a second feels broken. Tool calls have to be fast or covered with a natural filler.
• No visual channel. Everything must be said. Long lists and complex options do not work; the design has to be conversational.
• Injection through the caller. The person on the line is untrusted input, and social engineering is the natural attack: "as your supervisor, I authorize..."

What the harness does differently

• Confirm before acting on anything with a side effect, by reading it back: "I will cancel the reservation for the 14th. Is that right?"
• Tool calls under a latency budget, with a spoken acknowledgment if they will take a moment.
• A parallel text transcript for tracing, evals, and compliance, including what the model heard (the recognition) versus what was said.
• Escalation to a human as a first-class path, with the transcript handed over.
• Evals on recordings. Task completion, confirmation correctness, interruptions handled, and a judge on tone, run on real (consented) calls and on synthetic callers with scripted goals.

What They Share

Both are agents: a model proposing actions, a harness validating and executing them, a trace, an eval suite. Everything from harness engineering applies. The differences are a wider, less structured perception, a smaller and less scopable action space, and a harder observation problem for gating side effects. Design for those, and the rest of the agent playbook carries over.

Common Questions

What is a computer-use agent? An agent that operates a graphical interface the way a person does: it looks at a screenshot, decides where to click or what to type, acts, and looks again. It works with software that has no API, at the cost of being slower and more fragile than calling an API or a shell.

How do realtime voice agents differ from speech-to-text plus a chat model? Audio-native models process and generate speech directly with low latency and can handle interruptions, tone, and overlap, which the transcribe-then-respond pipeline cannot. The tradeoff is less control over the intermediate text and harder evaluation.

Are computer-use agents safe to run unattended? Not without the same containment as any agent: a sandboxed desktop or browser profile, no credentials the task does not need, approval gates for irreversible actions, and a trace of every action. A screenshot is untrusted content too; injection through what the agent sees on screen is real.

What to Practice Next

For a computer-use agent: put it in a fresh VM, give it a five-step task on a site you control, and write the side-effect gating rules by watching where it would have clicked something irreversible. For a voice agent: build a synthetic caller with a scripted goal and a twist (an interruption, a mumbled name) and grade twenty runs on confirmation correctness. The module AI Security for Agents: Prompt Injection, Excessive Agency, and Sandboxing covers the isolation and gating both need.