Multi-scale Transformer Language Models

Sandeep Subramanian, Ronan Collobert, Marc'Aurelio Ranzato +1

We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias…