Character-based NMT with Transformer

Abstract

Character-based translation has several appealing advantages, but its performance is in general worse than a carefully tuned BPE baseline. In this paper we study the impact of character-based input and output with the Transformer architecture. In particular, our experiments on EN-DE show that character-based Transformer models are more robust than their BPE counterpart, both when translating noisy text, and when translating text from a different domain. To obtain comparable BLEU scores in clean, in-domain data and close the gap with BPE-based models we use known techniques to train deeper Transformer models.

References in corpus (12)

• Distributed Representations of Words and Phrases and their Compositionality
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Residual Learning for Image Recognition
• LSTM-based Deep Learning Models for Non-factoid Answer Selection
• Character-based Neural Machine Translation
• Learning Deep Transformer Models for Machine Translation
• Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network
• Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
• Learning to Segment Inputs for NMT Favors Character-Level Processing
• Training Deeper Neural Machine Translation Models with Transparent Attention
• Domain Control for Neural Machine Translation
• Learning Distributed Representations of Texts and Entities from Knowledge Base