106 citations · 785 across the 59 of their papers we have counts for
11 papers · 1 filter
Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation
Nikolay Bogoychev, Rico Sennrich
The quality of neural machine translation can be improved by leveraging additional monolingual resources to create synthetic training data. Source-side monolingual data can be (for…
Domain Robustness in Neural Machine Translation
Mathias Müller, Annette Rios, Rico Sennrich
Translating text that diverges from the training domain is a key challenge for machine translation. Domain robustness---the generalization of models to unseen test domains---is low…
Root Mean Square Layer Normalization
Biao Zhang, Rico Sennrich
Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in ha…
Context-Aware Monolingual Repair for Neural Machine Translation
Elena Voita, Rico Sennrich, Ivan Titov
Modern sentence-level NMT systems often produce plausible translations of isolated sentences. However, when put in context, these translations may end up being inconsistent with ea…
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
Elena Voita, Rico Sennrich, Ivan Titov
We seek to understand how the representations of individual tokens and the structure of the learned feature space evolve between layers in deep neural networks under different lear…
Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention
Biao Zhang, Ivan Titov, Rico Sennrich
The general trend in NLP is towards increasing model capacity and performance via deeper neural networks. However, simply stacking more layers of the popular Transformer architectu…
Encoders Help You Disambiguate Word Senses in Neural Machine Translation
Gongbo Tang, Rico Sennrich, Joakim Nivre
Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of…
Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
Denis Emelin, Ivan Titov, Rico Sennrich
The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context…
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Elena Voita, David Talbot, Fedor Moiseev +2
Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by i…
A Lightweight Recurrent Network for Sequence Modeling
Biao Zhang, Rico Sennrich
Recurrent networks have achieved great success on various sequential tasks with the assistance of complex recurrent units, but suffer from severe computational inefficiency due to…
Revisiting Low-Resource Neural Machine Translation: A Case Study
Rico Sennrich, Biao Zhang
It has been shown that the performance of neural machine translation (NMT) drops starkly in low-resource conditions, underperforming phrase-based statistical machine translation (P…