Comparing Transformers and Hybrid Models at the Token Level

Yanhong Li, William Merrill

Hybrid language models that mix attention and recurrent layers have shown promise: theoretically, recurrent layers ameliorate the limitations of pure transformers on state tracking…