AbbIE: Autoregressive Block-Based Iterative Encoder for Efficient Sequence Modeling

Preslav Aleksandrov, Meghdad Kurmanji, Fernando Garcia Redondo +7

We introduce the Autoregressive Block-Based Iterative Encoder (AbbIE), a novel recursive generalization of the encoder-only Transformer architecture, which achieves better perplexi…