Separations in the Representational Capabilities of Transformers and Recurrent Architectures

Satwik Bhattamishra, Michael Hahn, Phil Blunsom +1

Transformer architectures have been widely adopted in foundation models. Due to their high inference costs, there is renewed interest in exploring the potential of efficient recurr…