Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs (opens in new tab)

This is a small study that explores using tool calls to wrap untrusted parts of prompts. OpenAI's model spec considers tool results the least trusted kind of input. If tool-wrapping helped, it would be an easy way to improve robustness while using existing APIs models already support. In 3 tested tasks it didn't seem to broadly help, and in some cases made things worse. We advocate for more understanding of the instruction hierarchy and ideas around better primitives for untrusted inputs. The...