Neutron: An Implementation of the Transformer Translation Model and its Variants

Abstract

The Transformer translation model is easier to parallelize and provides better performance compared to recurrent seq2seq models, which makes it popular among industry and research community. We implement the Neutron in this work, including the Transformer model and its several variants from most recent researches. It is highly optimized, easy to modify and provides comparable performance with interesting features while keeping readability.

References in corpus (8)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Searching for Activation Functions
• On the Variance of the Adaptive Learning Rate and Beyond
• Sockeye: A Toolkit for Neural Machine Translation
• THUMT: An Open Source Toolkit for Neural Machine Translation

Cited by in corpus (8)

• Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation
• Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers
• Neural Machine Translation: A Review and Survey
• Lipschitz Constrained Parameter Initialization for Deep Transformers
• UdS Submission for the WMT 19 Automatic Post-Editing Task
• Dynamically Adjusting Transformer Batch Size by Monitoring Gradient Direction Change
• Learning Hard Retrieval Decoder Attention for Transformers
• Learning Source Phrase Representations for Neural Machine Translation