An AI Feature Has No "Tests Pass" Moment. So I Write the Eval First. (opens in new tab)

I was building an "Ask This Book" feature: readers can ask questions about a book while they're reading it. One requirement sounded simple: A reader on chapter 3 must never receive spoilers from chapter 30. My first instinct was the same as everyone else's: tell the model not to spoil future chapters. Something like: "Please don't reveal information from chapters the reader hasn't reached yet." And honestly, it mostly worked. The problem is that "mostly" is useless. A user only needs one spoi...