Your AI Product Needs Evals – Hamel's Blog (opens in new tab)

Like software engineering, success with AI hinges on how fast you can iterate. You must have processes and tools for: 1. Evaluating quality (ex: tests). 2. Debugging issues (ex: logging & inspecting data). 3. Changing the behavior or the system (prompt eng, fine-tuning, writing code) Many people focus exclusively on #3 above, which prevents them from improving their LLM products beyond a demo.1 Doing all three activities well creates a virtuous cycle differentiating great from mediocre AI pro...