Untitled Draft (opens in new tab)

TLDR:Frontier models can detect when they're being evaluated and change their behavior, which risks compromising safety benchmarks. We introduce LURE (Live-Usage Replay Evaluations), a method that constructs alignment evals by replaying realistic conversations and appending a safety-relevant test at the end, rather than building evaluation scenarios from scratch, as other evals do. We instantiate it for scheming, AI safety research sabotage, and sycophancy. Measured against three black box ev...