An Empirical Exploration of Curriculum Learning for Neural Machine Translation

Abstract

Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better models faster. We adopt a probabilistic view of curriculum learning, which lets us flexibly evaluate the impact of curricula design, and perform an extensive exploration on a German-English translation task. Results show that it is possible to improve convergence time at no loss in translation quality. However, results are highly sensitive to the choice of sample difficulty criteria, curriculum schedule and other hyperparameters.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Practical recommendations for gradient-based training of deep architectures
• Sockeye: A Toolkit for Neural Machine Translation
• Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
• Boosting Neural Machine Translation
• A comprehensive study of batch construction strategies for recurrent neural networks in MXNet

Cited by in corpus (32)

• Self-Paced Learning for Neural Machine Translation
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Competence-based Curriculum Learning for Neural Machine Translation
• A Survey on Curriculum Learning
• Norm-Based Curriculum Learning for Neural Machine Translation
• Curriculum Learning for Domain Adaptation in Neural Machine Translation
• Do Data-based Curricula Work?
• Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation
• Learning a Multi-Domain Curriculum for Neural Machine Translation
• Confidence-Guided Learning Process for Continuous Classification of Time Series
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Optimizing Data Usage via Differentiable Rewards
• Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation
• Token-level Adaptive Training for Neural Machine Translation
• Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models
• Curriculum Learning Strategies for IR: An Empirical Study on Conversation Response Ranking
• Dynamically Composing Domain-Data Selection with Clean-Data Selection by "Co-Curricular Learning" for Neural Machine Translation
• Reinforcement Learning based Curriculum Optimization for Neural Machine Translation
• Curriculum Learning with Diversity for Supervised Computer Vision Tasks
• Word Interdependence Exposes How LSTMs Compose Representations
• Statistical Measures For Defining Curriculum Scoring Function
• Exploiting Curriculum Learning in Unsupervised Neural Machine Translation
• Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits
• Self-Guided Curriculum Learning for Neural Machine Translation
• Towards Multimodal Response Generation with Exemplar Augmentation and Curriculum Optimization
• Dynamic Curriculum Learning for Low-Resource Neural Machine Translation
• Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning
• LSTMs Compose (and Learn) Bottom-Up
• Learning Policies for Multilingual Training of Neural Machine Translation Systems
• Token-wise Curriculum Learning for Neural Machine Translation
• Data Ordering Patterns for Neural Machine Translation: An Empirical Study
• Simple and Effective Input Reformulations for Translation