Quoting Boaz Barak, Gabriel Wu, Jeremy Chen and Manas Joglekar (opens in new tab)

<blockquote cite="https://alignment.openai.com/confessions/"><p>When we optimize responses using a reward model as a proxy for “goodness” in reinforcement learning, models sometimes learn to “hack” this proxy and output an answer that only “looks good” to it (because coming up with an answer that is actually good can be hard). The philosophy behind confessions is that we can train models to produce a second output — aka a “confession” — that is rewarded solely for honesty, which we will ...