2.2k citations · 5k across the 31 of their papers we have counts for
36 papers
Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
Piotr Padlewski, Max Bain, Matthew Henderson +19
We introduce Vibe-Eval: a new open benchmark and framework for evaluating multimodal chat models. Vibe-Eval consists of 269 visual understanding prompts, including 100 of hard diff…
Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
Reka Team, Aitor Ormazabal, Che Zheng +23
We introduce Reka Core, Flash, and Edge, a series of powerful multimodal language models trained from scratch by Reka. Reka models are able to process and reason with text, images,…
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Yi Tay, Mostafa Dehghani, Samira Abnar +7
There have been a lot of interest in the scaling properties of Transformer models. However, not much has been done on the front of investigating the effect of scaling properties of…
Questions Are All You Need to Train a Dense Passage Retriever
Devendra Singh Sachan, Mike Lewis, Dani Yogatama +3
We introduce ART, a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. Dense retrieval is a central challen…
Emergent Abilities of Large Language Models
Jason Wei, Yi Tay, Rishi Bommasani +13
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictabl…
Language Models Can See: Plugging Visual Controls in Text Generation
Yixuan Su, Tian Lan, Yahui Liu +5
Generative language models (LMs) such as GPT-2/3 can be prompted to generate text with remarkable quality. While they are designed for text-prompted generation, it remains an open…
High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning
Paul Pu Liang, Yiwei Lyu, Xiang Fan +6
Many real-world problems are inherently multimodal, from spoken language, gestures, and paralinguistics humans use to communicate, to force, proprioception, and visual sensors on r…
A Contrastive Framework for Neural Text Generation
Yixuan Su, Tian Lan, Yan Wang +3
Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models o…
Relational Memory Augmented Language Models
Qi Liu, Dani Yogatama, Phil Blunsom
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve r…
Balancing Average and Worst-case Accuracy in Multitask Learning
Paul Michel, Sebastian Ruder, Dani Yogatama
When training and evaluating machine learning models on a large number of tasks, it is important to not only look at average task accuracy -- which may be biased by easy or redunda…
ABC: Attention with Bounded-memory Control
Hao Peng, Jungo Kasai, Nikolaos Pappas +5
Transformer architectures have achieved state-of-the-art results on a variety of sequence modeling tasks. However, their attention mechanism comes with a quadratic complexity in se…
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Yi Tay, Mostafa Dehghani, Jinfeng Rao +7
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come…
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering
Devendra Singh Sachan, Siva Reddy, William Hamilton +2
We present an end-to-end differentiable training method for retrieval-augmented open-domain question answering systems that combine information from multiple retrieved documents wh…
Random Feature Attention
Hao Peng, Nikolaos Pappas, Dani Yogatama +3
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at e…
Finetuning Pretrained Transformers into RNNs
Jungo Kasai, Hao Peng, Yizhe Zhang +6
Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's c…
Adaptive Semiparametric Language Models
Dani Yogatama, Cyprien de Masson d'Autume, Lingpeng Kong
We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our…
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya +11
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contra…
Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried +4
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests…
A Call for More Rigor in Unsupervised Cross-lingual Learning
Mikel Artetxe, Sebastian Ruder, Dani Yogatama +2
We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them. An existing rationale…
Modelling Latent Skills for Multitask Language Generation
Kris Cao, Dani Yogatama
We present a generative model for multitask conditional language generation. Our guiding hypothesis is that a shared set of latent skills underlies many disparate language generati…
A Mutual Information Maximization Perspective of Language Representation Learning
Lingpeng Kong, Cyprien de Masson d'Autume, Wang Ling +3
We show state-of-the-art word representation learning methods maximize an objective function that is a lower bound on the mutual information between different parts of a word seque…
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Po-Sen Huang, Huan Zhang, Ray Jiang +6
Advances in language modeling architectures and the availability of large text corpora have driven progress in automatic text generation. While this results in models capable of ge…
On the Cross-lingual Transferability of Monolingual Representations
Mikel Artetxe, Sebastian Ruder, Dani Yogatama
State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been…
Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagation
Po-Sen Huang, Robert Stanforth, Johannes Welbl +5
Neural networks are part of many contemporary NLP systems, yet their empirical successes come at the price of vulnerability to adversarial attacks. Previous work has used adversari…