PointerBench finds GUI agents miss precise targets despite strong general screenshot scores

PointerBench was built to isolate precise graphical targeting from the broader claim that an agent can perform an office workflow. Its 1,500 screenshot-instruction pairs cover spreadsheet cells, character-level document text and controls drawn from 100 professional applications, with coordinates evaluated against a fixed 1024-by-768 image contract. Unlike a live workflow benchmark, it does not give the model the authority to click, send, pay or edit a real account. Benchmark paper.

That distinction changes how the scores read. The authors report 83.2% overall accuracy and 69.6% text-target accuracy for Fable 5, while GPT-5.6 Sol achieved 77.7% overall and 53.4% on text. Text selection is markedly more demanding than locating a large button. The study also shows that raw performance for some other model families changed substantially once their output coordinates were normalized to the benchmark's expected frame. A wrong coordinate protocol and a wrong visual target are different failure modes; the harness must not quietly count one as the other.

The practical side effect remains unmeasured. Selecting an adjacent character in a draft can corrupt content, while selecting an adjacent approval or delete control may execute an unrelated operation. The benchmark reports geometric correctness, not the probability or severity of those effects, and does not verify whether the model had authorization to act on the chosen target. Treat the result as evidence about visual grounding and harness validity, then test live workflows separately with replayable state changes and explicit permission boundaries.

The next evaluation should record the selected coordinate, target identity, execution permission and actual post-click state, using the same scale and frame for every compared model. Without that chain, a high screenshot percentage is neither a safe-operation score nor a justification for expanding agent authority.