Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation
arXiv:1811.01116
Abstract
We aim to better exploit the limited amounts of parallel text available in low-resource settings by introducing a differentiable reconstruction loss for neural machine translation (NMT). This loss compares original inputs to reconstructed inputs, obtained by back-translating translation hypotheses into the input language. We leverage differentiable sampling and bi-directional NMT to train models end-to-end, without introducing additional parameters. This approach achieves small but consistent BLEU improvements on four language pairs in both translation directions, and outperforms an alternative differentiable reconstruction strategy based on hidden states.
arXiv commentAccepted at NAACL 2019
References in corpus (8)
- Categorical Reparameterization with Gumbel-Softmax
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Dual Learning for Machine Translation
- Layer Normalization
- GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
- Sockeye: A Toolkit for Neural Machine Translation
- Translating Pro-Drop Languages with Reconstruction Models