TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling

Tze Yuang Chong, Xuyang Wang, Lin Yang +1

In this paper, we describe the use of recurrent neural networks to capture sequential information from the self-attention representations to improve the Transformers. Although self…