Transformer Quality in Linear Time

Weizhe Hua, Zihang Dai, Hanxiao Liu +1

We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention un…