Computer-Use Agents: The 60-Point Gap Between Demo and Production

In the demo video the agent opens a browser, fills the form, downloads the file, and it all takes forty seconds. Put the same agent on a real workflow and it gets lost around step three hundred. That is not a sloppy implementation; it is a measured, published pattern. The gap between two 2026 benchmark results gives you the single most useful frame available on computer-use agents. This article covers that gap, the loop an agent actually runs, where grounding breaks, and when you should reach for plain automation instead.

Two numbers: 83.5% and 20.6%#

A computer-use agent looks at a screenshot and emits mouse and keyboard actions — it operates a computer the way a person does, rather than through an API.

The reference benchmark here is OSWorld. Its verified first version is effectively saturated: the best result is 83.5%, against a human baseline of 72.36% on the same set. Read only that, and you would happily conclude that agents have passed humans.

OSWorld 2.0 (arXiv:2606.29537, submitted 28 June 2026) was built to test exactly that claim: 108 long-horizon workflows, a median human completion time of about 1.6 hours, and an average of 318 tool calls per task. The best frontier agent completes 20.6% of them at a 500-step budget, with a 54.8% partial score.

The 60-point gap is not a model defect — it is the difference between a short task and a long one.

The practical consequence: a demo showing an agent finish in three steps inside one application is no evidence at all that it can carry a three-hundred-step process. The right procurement question is not “what did it score” but “at what step budget was it measured”.

The loop: perceive, ground, act, verify#

Every computer-use agent turns the same four-step loop. Conceptually:

while not done and step < step_budget:
    frame  = take_screenshot()               # 1. perceive
    plan   = model(frame, goal, history)     # 2. ground + decide
    result = execute(plan.action)            # 3. act (click/type/scroll)
    done   = verify(plan, take_screenshot()) # 4. verify
    step += 1

The cost of this loop surprises most teams. Every step means at least one screenshot; every screenshot means image tokens; a 318-step task means the same window entering context dozens of times. That is why serious deployments do three things as a matter of course: drop stale screenshots from context, send downscaled rather than full-resolution frames, and impose a step budget.

Pixels or the accessibility tree?#

Grounding — turning an intent like “the save button” into a coordinate — is the most brittle part of the system. Two approaches:

• Pixel-based: the model sees only the screenshot and emits (x, y). It works everywhere — games, remote desktops, canvas UIs — and is fragile to resolution, DPI scaling and theme changes.
• Accessibility-tree or DOM based: the model reads a structural representation and selects an element by identity. Far more stable and verifiable, but only available where the application publishes that tree properly.

What works in production is usually hybrid: structural selectors where available, pixels as the fallback. The load-bearing piece is verification — after an action, confirm with a separate check that the expected change actually happened. An agent that never distinguishes “I clicked” from “the click worked” will carry its mistake for another three hundred steps.

Where locale breaks grounding#

Text matching against on-screen labels is where locale quietly destroys an agent, and Turkish is the textbook case. Standard case-folding is wrong for Turkish, and you can verify the behaviour directly in Python:

>>> "I".lower()          # Turkish expects the dotless 'ı'
'i'
>>> "İ".lower()          # returns TWO characters, not one
'i̇'                # 'i' + U+0307 combining dot above
>>> len("İ".lower())
2
>>> "ISTANBUL".lower()   # not 'ıstanbul'
'istanbul'

So matching a button labelled “İPTAL” with text.lower() == "iptal" fails because of an invisible combining character. The fix is a locale-aware fold followed by Unicode normalisation:

import unicodedata

def tr_fold(s: str) -> str:
    """Lowercase Turkish UI text safely and normalise the result."""
    s = s.replace("I", "ı").replace("İ", "i")   # I->ı, İ->i
    s = s.lower()
    return unicodedata.normalize("NFC", s)

assert tr_fold("İPTAL")    == "iptal"
assert tr_fold("ISTANBUL") == "ıstanbul"

Two related traps sit next to it. Keyboard layout: punctuation lives in different places on Turkish Q and Turkish F layouts, so text entry must go through a type/insert action rather than raw key codes. And date and number formats: the screen shows 13.08.2026 and 1.250,75, and a verification step that parses those with English assumptions silently produces the wrong answer.

One hard boundary: do not automate identity-bound government or banking flows with a pixel agent. Those actions are irreversible, tied to a real identity, and frequently prohibited by the service terms. There is no undo button when an agent clicks the wrong one.

Security: the unsolved part#

A computer-use agent by definition looks at untrusted content and then acts. That is ideal ground for indirect prompt injection: hidden text on a page reads to the agent as an instruction, and the agent follows it. OpenAI stated in December 2025 that prompt injection is “unlikely to ever be fully solved”, and Brave demonstrated indirect injection against Perplexity Comet using hidden page elements.

The correct reading is not “don’t use agents” but “narrow what the agent can do”. The boundaries that work in production: run the agent in a dedicated VM or container; restrict egress with an allowlist of the domains it genuinely needs; keep session cookies and password managers out of the profile the agent uses; and require human approval for every irreversible action. No text the agent reads is an instruction to the agent.

Agent or Playwright?#

The most expensive mistake is handing a deterministic job to an agent. The decision table:

In practice the sturdiest architecture is hybrid: let the agent discover the path, then compile that path into a deterministic script. Sending the agent out to rediscover it on every run is both the most expensive and the most fragile option.

Computer-use agents do add a genuine capability, but the 60-point gap between a short demo and a long workflow is the number your architecture should be built around. For the shape of the loop itself see autonomous agent architectures, for why on-screen text must never be treated as an instruction see prompt injection and jailbreaks, and to find out whether your agent actually works see agent evaluation.