Language Modeling with Learned Meta-Tokens

Alok N. Shah, Khush Gupta, Keshav Ramji +1

While modern Transformer-based language models (LMs) have achieved major success in multi-task generalization, they often struggle to capture long-range dependencies within their c…