Simple Recurrence Improves Masked Language Models

Tao Lei, Ran Tian, Jasmijn Bastings +1

In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into t…