Massive Exploration of Neural Machine Translation Architectures

Abstract

Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results.

9 pages, 2 figures, 8 tables, submitted to ACL 2017, open source code at https://github.com/google/seq2seq/

References in corpus (2)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Machine Translation with Reconstruction

Cited by in corpus (15)

• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Learning Deep Transformer Models for Machine Translation
• Neural Text Generation: A Practical Guide
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Disentangled Non-Local Neural Networks
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Multi-node Bert-pretraining: Cost-efficient Approach
• Teaching Machines to Converse
• Skeleton based Activity Recognition by Fusing Part-wise Spatio-temporal and Attention Driven Residues
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
• Structure-Aware Generation Network for Recipe Generation from Images
• Retrosynthetic reaction prediction using neural sequence-to-sequence models
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation