Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks

Paul Suganthan, Fedor Moiseev, Le Yan +7

Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language proc…