2.2k citations · 5k across the 31 of their papers we have counts for
8 papers · 1 filter
Balancing Average and Worst-case Accuracy in Multitask Learning
Paul Michel, Sebastian Ruder, Dani Yogatama
When training and evaluating machine learning models on a large number of tasks, it is important to not only look at average task accuracy -- which may be biased by easy or redunda…
ABC: Attention with Bounded-memory Control
Hao Peng, Jungo Kasai, Nikolaos Pappas +5
Transformer architectures have achieved state-of-the-art results on a variety of sequence modeling tasks. However, their attention mechanism comes with a quadratic complexity in se…
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Yi Tay, Mostafa Dehghani, Jinfeng Rao +7
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come…
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
Devendra Singh Sachan, Siva Reddy, William Hamilton +2
We present an end-to-end differentiable training method for retrieval-augmented open-domain question answering systems that combine information from multiple retrieved documents wh…
Random Feature Attention
Hao Peng, Nikolaos Pappas, Dani Yogatama +3
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at e…
Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang +6
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's c…
Adaptive Semiparametric Language Models
Dani Yogatama, Cyprien de Masson d'Autume, Lingpeng Kong
We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our…
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya +11
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contra…