Massive Exploration of Neural Machine Translation Architectures

Abstract

Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework that enables researchers to easily experiment with novel techniques and reproduce state of the art results.

arXiv comment9 pages, 2 figures, 8 tables, submitted to ACL 2017, open source code at https://github.com/google/seq2seq/

References in corpus (8)

• TensorFlow: A system for large-scale machine learning
• LSTM: A Search Space Odyssey
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Densely Connected Convolutional Networks
• Speed/accuracy trade-offs for modern convolutional object detectors
• Modeling Coverage for Neural Machine Translation
• A Convolutional Encoder Model for Neural Machine Translation
• Neural Machine Translation with Reconstruction

Cited by in corpus (40)

• Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
• Improving Text-to-SQL Evaluation Methodology
• NeMo: a toolkit for building AI applications using Neural Modules
• Supervised Contrastive Learning
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Learning Deep Transformer Models for Machine Translation
• OpenNMT: Neural Machine Translation Toolkit
• Relation Networks for Object Detection
• Towards Binary-Valued Gates for Robust LSTM Training
• An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization
• Neural Text Generation: A Practical Guide
• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
• Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Automatic Article Commenting: the Task and Dataset
• Disentangled Non-Local Neural Networks
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Multi-node Bert-pretraining: Cost-efficient Approach
• Skeleton based Activity Recognition by Fusing Part-wise Spatio-temporal and Attention Driven Residues
• Exploring Hyper-Parameter Optimization for Neural Machine Translation on GPU Architectures
• Teaching Machines to Converse
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• Learning-based Prediction and Uplink Retransmission for Wireless Virtual Reality (VR) Network
• Attentive Sequence-to-Sequence Learning for Diacritic Restoration of Yorùbá Language Text
• Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation
• Transfer Learning in Electronic Health Records through Clinical Concept Embedding
• What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding
• Structure-Aware Generation Network for Recipe Generation from Images
• Meta-Embeddings Based On Self-Attention
• RISC-NN: Use RISC, NOT CISC as Neural Network Hardware Infrastructure
• Retrosynthetic reaction prediction using neural sequence-to-sequence models
• Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Domain Adaptation of NMT models for English-Hindi Machine Translation Task at AdapMT ICON 2020
• Handling Homographs in Neural Machine Translation
• The emergent algebraic structure of RNNs and embeddings in NLP
• DeepDrawing: A Deep Learning Approach to Graph Drawing
• An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation