1.3k citations · 3.7k across the 43 of their papers we have counts for
5 papers · 1 filter
Transformer Feed-Forward Layers Are Key-Value Memories
Mor Geva, Roei Schuster, Jonathan Berant +1
Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer…
The Turking Test: Can Language Models Understand Instructions?
Avia Efrat, Omer Levy
Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in n…
Neural Machine Translation without Embeddings
Uri Shaham, Omer Levy
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is t…
Aligned Cross Entropy for Non-Autoregressive Machine Translation
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer +1
Non-autoregressive machine translation models significantly speed up decoding by allowing for parallel prediction of the entire target sequence. However, modeling word order is mor…
Semi-Autoregressive Training Improves Mask-Predict Decoding
Marjan Ghazvininejad, Omer Levy, Luke Zettlemoyer
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right appr…