HRM-Text: Efficient Pretraining Beyond Scaling (opens in new tab)

URL Source: Markdown Content: Guan Wang 1,∗,†, Changling Liu 1,∗, Chenyu Wang 2, Cai Zhou 2, Yuhao Sun 1, Yifei Wu 1, Shuai Zhen 1, Luca Scimeca 1, Yasin Abbasi Yadkori 1,† 1 Sapient Intelligence 2 MIT ###### Abstract The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foundational research. In contrast, biological systems demonstrate highly sample-efficient learning through multi-timescale p...