310 citations · 486 across the 8 of their papers we have counts for
14 papers
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokens
Zhanpeng Zeng, Cole Hawkins, Mingyi Hong +4
Transformers are central in modern natural language processing and computer vision applications. Despite recent works devoted to reducing the quadratic cost of such models (as a fu…
Modeling Context With Linear Attention for Scalable Document-Level Translation
Zhaofeng Wu, Hao Peng, Nikolaos Pappas +1
Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations. However, these models, predominantly based on transfo…
Sentence Bottleneck Autoencoders from Transformer Language Models
Ivan Montero, Nikolaos Pappas, Noah A. Smith
Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast…
Random Feature Attention
Hao Peng, Nikolaos Pappas, Dani Yogatama +3
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at e…
Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang +6
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's c…
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas +1
Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long se…