MeSH: Memory-as-State-Highways for Recursive Transformers

Chengting Yu, Xiaobo Shu, Yadao Wang +8

Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth. However, under matched compute, recursive mode…