Olmo Hybrid: From Theory to Practice and Back

William Merrill, Yanhong Li, Tyler Romero +19

Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attentio…