Learning to Segment Inputs for NMT Favors Character-Level Processing

Abstract

Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths, hence the modeling depth, and source and target vocabularies, which in turn determine model size, computational costs of softmax normalization, and handling of out-of-vocabulary words. However, the current practice is to use static, heuristic-based segmentations that are fixed before NMT training. This begs the question whether the chosen segmentation is optimal for the translation task. To overcome suboptimal segmentation choices, we present an algorithm for dynamic segmentation based on the Adaptative Computation Time algorithm (Graves 2016), that is trainable end-to-end and driven by the NMT objective. In an evaluation on four translation tasks we found that, given the freedom to navigate between different segmentation levels, the model prefers to operate on (almost) character level, providing support for purely character-level NMT models from a novel angle.

arXiv commentTechnical report for IWSLT 2018 paper

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Adaptive Computation Time for Recurrent Neural Networks
• An Actor-Critic Algorithm for Sequence Prediction
• Character-based Neural Machine Translation
• Sockeye: A Toolkit for Neural Machine Translation
• An Efficient Character-Level Neural Machine Translation

Cited by in corpus (10)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Controlling the Output Length of Neural Machine Translation
• A Latent Morphology Model for Open-Vocabulary Neural Machine Translation
• Character-based NMT with Transformer
• Adapting Multilingual Neural Machine Translation to Unseen Languages
• The boundaries of meaning: a case study in neural machine translation
• Controlling Utterance Length in NMT-based Word Segmentation with Attention
• Adversarial Subword Regularization for Robust Neural Machine Translation
• Vocabulary Learning via Optimal Transport for Neural Machine Translation
• Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation