Let's reproduce GPT-2 (124M) (opens in new tab)

<iframe id="ytplayer" type="text/html" width="640" height="360" src="https://www.youtube-nocookie.com/embed/l8pRSuU81PU" frameborder="0" allowfullscreen="" referrerpolicy="strict-origin-when-cross-origin"></iframe><br><span style="white-space: pre-wrap;">We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training...</span>