607 citations · 891 across the 9 of their papers we have counts for
5 papers · 1 filter
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis, Yinhan Liu, Naman Goyal +5
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a…
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu, Myle Ott, Naman Goyal +7
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often…
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Mandar Joshi, Danqi Chen, Yinhan Liu +3
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather…
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Marjan Ghazvininejad, Omer Levy, Yinhan Liu +1
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of…
Cloze-driven Pretraining of Self-attention Networks
Alexei Baevski, Sergey Edunov, Yinhan Liu +2
We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our mo…