241 citations · 563 across the 9 of their papers we have counts for
17 papers
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru +15
Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few…
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
Peter Hase, Mona Diab, Asli Celikyilmaz +5
Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decou…
Multi-task Retrieval for Knowledge-Intensive Tasks
Jean Maillard, Vladimir Karpukhin, Fabio Petroni +4
Retrieving relevant contexts from a large corpus is a crucial step for tasks such as open-domain question answering and fact checking. Although neural retrieval outperforms traditi…
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
Jingfei Du, Myle Ott, Haoran Li +2
The state of the art on many NLP tasks is currently achieved by large pre-trained language models, which require a considerable amount of computation. We explore a setting where ma…
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Wenhan Xiong, Jingfei Du, William Yang Wang +1
Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition…
SemEval-2013 Task 2: Sentiment Analysis in Twitter
Preslav Nakov, Zornitsa Kozareva, Alan Ritter +3
In recent years, sentiment analysis in social media has attracted a lot of research interest and has been used for a number of applications. Unfortunately, research has been hinder…
SemEval-2014 Task 9: Sentiment Analysis in Twitter
Sara Rosenthal, Preslav Nakov, Alan Ritter +1
We describe the Sentiment Analysis in Twitter task, ran as part of SemEval-2014. It is a continuation of the last year's task that ran successfully as part of SemEval-2013. As in 2…
SemEval-2015 Task 10: Sentiment Analysis in Twitter
Sara Rosenthal, Saif M Mohammad, Preslav Nakov +3
In this paper, we describe the 2015 iteration of the SemEval shared task on Sentiment Analysis in Twitter. This was the most popular sentiment analysis shared task to date with mor…
SemEval-2016 Task 4: Sentiment Analysis in Twitter
Preslav Nakov, Alan Ritter, Sara Rosenthal +2
This paper discusses the fourth year of the ``Sentiment Analysis in Twitter Task''. SemEval-2016 Task 4 comprises five subtasks, three of which represent a significant departure fr…
Unsupervised Cross-lingual Representation Learning at Scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal +7
This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transfor…
Emerging Cross-lingual Structure in Pretrained Language Models
Shijie Wu, Alexis Conneau, Haoran Li +2
We study the problem of multilingual masked language modeling, i.e. the training of a single model on concatenated text from multiple languages, and present a detailed study of sev…
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis, Yinhan Liu, Naman Goyal +5
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a…
Bridging the domain gap in cross-lingual document classification
Guokun Lai, Barlas Oguz, Yiming Yang +1
The scarcity of labeled training data often prohibits the internationalization of NLP models to multiple languages. Recent developments in cross-lingual understanding (XLU) has mad…
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu, Myle Ott, Naman Goyal +7
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often…
Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
Angli Liu, Jingfei Du, Veselin Stoyanov
Traditional language models are unable to efficiently model entity names observed in text. All but the most popular named entities appear infrequently in text providing insufficien…
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott +4
State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a si…
Simple Fusion: Return of the Language Model
Felix Stahlberg, James Cross, Veselin Stoyanov
Neural Machine Translation (NMT) typically leverages monolingual data in training through backtranslation. We investigate an alternative simple method to use monolingual data for N…