Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory

Runxi Cheng, Yuchen Guan, Yongxian Wei +7

Scaling conditional memory offers a promising way to increase language-model capacity, but existing methods such as Engram learn large memory tables from scratch during pre-trainin…