1.3k citations · 3.7k across the 43 of their papers we have counts for
13 papers · 1 filter
Improving Transformer Models by Reordering their Sublayers
Ofir Press, Noah A. Smith, Omer Levy
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers. Could ordering the sublayers in a different pattern lead to better performance? We…
Blockwise Self-Attention for Long Document Understanding
Jiezhong Qiu, Hao Ma, Omer Levy +3
We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the…
Generalization through Memorization: Nearest Neighbor Language Models
Urvashi Khandelwal, Omer Levy, Dan Jurafsky +2
We introduce NN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a -nearest neighbors (NN) model. The nearest neighbors are com…
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Mike Lewis, Yinhan Liu, Naman Goyal +5
We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a…
Structural Language Models of Code
Uri Alon, Roy Sadaka, Omer Levy +1
We address the problem of any-code completion - generating a missing piece of source code in a given program without any restriction on the vocabulary or structure. We introduce a…
BERT for Coreference Resolution: Baselines and Analysis
Mandar Joshi, Omer Levy, Daniel S. Weld +1
We apply BERT to coreference resolution, achieving strong improvements on the OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. A qualitative analysis of model predictions indicat…