DiffusionBlocks: Residual Networks Are Secretly Diffusion Models (opens in new tab)

What if you could train a deep network one slice at a time, with no backpropagation running through the whole thing? Here is a fact that has quietly shaped the last decade of AI: to train a network with backpropagation, you have to remember everything. Every layer’s output, every intermediate activation, all of it kept alive in memory from the forward pass so the backward pass can use it to compute gradients. Train a 100-layer model and you pay for 100 layers’ worth of activations at once. Me...