Published on October 5, 2025 4:53 PM GMT

TLDR: A new paper from Simular Research using a new scaling technique for computer-use agents achieved 70% on OSWorld-Verified, a benchmark for computer use. A skilled human scores 72% on this benchmark. Their new technique, Behavior Best-of-N (bBoN), shows promise for improving performance of agents, perhaps on a variety of tasks, though new research will be needed to test this.

One current bottleneck for long-running agentic AI is the inability to perform complex computer use with high accuracy. The most widely recognized benchmark for complex computer use is the OSWorld-Verified. It tests a model on a variety of computer use tasks based on real-world examples. Examples include checking emails, performing tasks in Excel, and deleting cookies. There are many more types of tasks than these, but I think it gives a general picture of what the benchmark is measuring.

Without much fanfare, Simular Research has built a new agentic framework, powered by GPT-5, that achieved a new high score on this benchmark. At the end of September 2025, Claude 4.5 Sonnet was SOTA on this benchmark, achieving 61%. A skilled human on average scores 72% on OSWorld-Verified. Simular Research’s agent s3 w/ GPT-5 bBoN (N=10) is knocking at the door of this score, achieving 70% on October 2nd. They were able to achieve this breakthrough with a new technique, which they call Behavior Best-of-N. 

First, the implications of this parity seem pretty large. It should improve the reliability of agents pretty substantially, and the time horizon of tasks that agents can perform should grow. Performance will only improve from here. A few weeks or a few months from now, whatever it may be, AI should be able to perform computer use tasks with more accuracy than a skilled human, unless we have achieved saturation on this benchmark. This should eventually have implications for jobs. There are plenty of white collar jobs that are substantially just about computer use. Perhaps AI will need to be much more accurate than a human for replacement to happen, but that doesn’t seem particularly far off either. There could be diminishing returns, but it wouldn’t take much more improvement to achieve startling results.

Second, the new technique developed by Simular and laid out in this paper seems like a legitimate breakthrough. The intuition behind Behavior Best-of-N is that models are pretty inconsistent at certain tasks, sometimes they will nail it in one shot, and sometimes they will do something catastrophically wrong. However, it might be easier for a model to judge between many outcomes generated by other instances of models than it would be for any one instance to get the right answer confidently on its own. The paper describes it much better than I can:

To address these challenges, we introduce Behavior Best-of-N (bBoN), a novel framework that enables wide scaling of CUAs. Our approach first converts raw trajectories into behavior narratives: concise summaries that capture what the agent actually did and how it affected the environment, preserving task-relevant action–effect summaries while filtering away irrelevant detail at individual steps. These narratives provide a compact yet faithful representation that makes it easier for a judge to compare candidates. bBoN then performs selection directly over narratives, enabling reliable selection among multiple rollouts. In addition, we build upon existing CUAs and introduce an improved baseline computer-use agentic framework to generate high quality trajectories for bBoN.

This approach is intuitive and highly replicable. The newest component of this technique is the behavior narratives; before this, it was hard to use Best-of-N techniques with complex, agentic tasks because the output of such a task would not be legible. This is also possibly replicable for other complex tasks that have outputs that are traditionally not legible to LLMs. To be confident in this, there would need to be more research.  

I imagine that we will see a lot of agent architecture approaches like this in the future. OpenAI appears to be ahead of the curve here. In OpenAI’s successful ICPC entry, a coding competition, in September 2025, they outperformed all human and AI competitors using a Best-of-N approach. They described their technique like this:

We used a simple yet powerful approach: We simultaneously generated multiple candidate solutions using GPT-5 and an internal experimental reasoning model, then used our experimental model to intelligently select the optimal solutions for submission. There was no complex strategy or scaffolding.

Our models competed under the same conditions as human participants, including identical hidden test cases, time limits, memory constraints, hardware specifications, and sandboxed evaluation environments.

This sounds like bBoN without the novel behavior narratives component, making it closer to existing Best-of-N approaches. However, that still gives us another data point about the efficacy of Best-of-N techniques: it can achieve excellent results in solving hard coding problems. I’m curious to see what the results of a model like this would be on a benchmark like the SWEbench-Verified. Intuitively, I imagine the performance would be improved. The ICPC performance could be evidence of this; however, it could also be explained by their experimental reasoning model. Claude Code’s subagents seem to do something similar to bBoN, though I haven’t used it and don’t know if it’s truly comparable or how much it improves task accuracy.

Behavior Best-of-N certainly represents a breakthrough for AI computer use capabilities. If it turns out to be broadly generalizable to other complex tasks, it could be even more significant than that.