The Bitter Lesson’s Bitter Lesson (opens in new tab)

Originally published on Andrew’s SubStack. Richard Sutton and Dwarkesh discussed the Bitter Lesson, where Richard argued that babies and animals don’t learn through imitation, so state-of-the-art LLMs are pursuing the wrong path by imitating humans through next-token prediction. While Richard is correct about babies and animals, he overlooks the extraordinary computational savings we achieve through this […]