How to Score a User Simulator: Introducing USR-8 (opens in new tab)

Authors: José Santos, Shuo Qiu, Morteza Ziyadi User simulators have become a standard part of the agent-building toolkit: cheaper than real-user pilots, faster than scripted dialogues, and the only way to push an agent through hundreds of plausible conversations between every code change. The hard question is the one that comes next: how do you know the simulator itself is any good? It's easy to underestimate how much can go wrong. A simulator that ends every turn with "Thanks so much for you...