The Man Who Summoned Ghosts | Chapter 2: The Training Stack Is Not a Secret (opens in new tab)

From OpenAI to nanoGPT: why the training stack should feel legible, not magical. Originally published on Lei Hua's Substack. Anchors: 2023-05-23 · State of GPT @ Microsoft Build · 2023-11-23 · [1hr Talk] Intro to Large Language Models · Epigraph "99% of the compute is in pretraining. ... For applications, you want low-stakes things, with humans in the loop. Treat these models like cognitive interns." — Andrej Karpathy, State of GPT · 2023-05 The Return In the last two months of 2022, three th...