Lattice-Based Transformer Encoder for Neural Machine Translation

Abstract

Neural machine translation (NMT) takes deterministic sequences for source representations. However, either word-level or subword-level segmentations have multiple choices to split a source sequence with different word segmentors or different subword vocabulary sizes. We hypothesize that the diversity in segmentations may affect the NMT performance. To integrate different segmentations with the state-of-the-art NMT model, Transformer, we propose lattice-based encoders to explore effective word or subword representation in an automatic way during training. We propose two methods: 1) lattice positional encoding and 2) lattice-aware self-attention. These two methods can be used together and show complementary to each other to further improve translation performance. Experiment results show superiorities of lattice-based encoders in word-level and subword-level representations over conventional Transformer encoder.

Accepted by ACL 2019

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Character-based Neural Machine Translation
• Lattice-Based Recurrent Neural Network Encoders for Neural Machine Translation
• Syntax-Directed Attention for Neural Machine Translation
• Neural Word Segmentation Learning for Chinese
• Chinese Word Segmentation: Another Decade Review (2007-2017)
• Neural Lattice-to-Sequence Models for Uncertain Inputs
• Fast and Accurate Neural Word Segmentation for Chinese

Cited by in corpus (5)

• A Survey of Deep Learning Techniques for Neural Machine Translation
• Explicit Contextual Semantics for Text Comprehension
• Relative Positional Encoding for Transformers with Linear Complexity
• Hierarchical Contextualized Representation for Named Entity Recognition
• Confusion2vec 2.0: Enriching Ambiguous Spoken Language Representations with Subwords