Generating Long Sequences with Sparse Transformers

Rewon Child, Scott Gray, Alec Radford +1

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the at…