Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs

Hyunji Lee, Joykirat Singh, Zaid Khan +5

Recurrent-attention hybrid language models (LMs), which interleave attention and recurrent layers, are increasingly used to combine the efficiency of the recurrent layers with the…