Competence-based Curriculum Learning for Neural Machine Translation

Abstract

Current state-of-the-art NMT systems use large neural networks that are not only slow to train, but also often require many heuristics and optimization tricks, such as specialized learning rate schedules and large batch sizes. This is undesirable as it requires extensive hyperparameter tuning. In this paper, we propose a curriculum learning framework for NMT that reduces training time, reduces the need for specialized heuristics or large batch sizes, and results in overall better performance. Our framework consists of a principled way of deciding which training samples are shown to the model at different times during training, based on the estimated difficulty of a sample and the current competence of the model. Filtering training samples in this manner prevents the model from getting stuck in bad local optima, making it converge faster and reach a better solution than the common approach of uniformly sampling training examples. Furthermore, the proposed method can be easily applied to existing NMT models by simply modifying their input data pipelines. We show that our framework can help improve the training time and the performance of both recurrent neural network models and Transformers, achieving up to a 70% decrease in training time, while at the same time obtaining accuracy improvements of up to 2.2 BLEU.

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• On the Convergence of Adam and Beyond
• Training Tips for the Transformer Model
• Curriculum Learning and Minibatch Bucketing in Neural Machine Translation
• An Empirical Exploration of Curriculum Learning for Neural Machine Translation
• Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation

Cited by in corpus (10)

• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Dynamic Data Selection and Weighting for Iterative Back-Translation
• Optimizing Data Usage via Differentiable Rewards
• Gradual Domain Adaptation in the Wild:When Intermediate Distributions are Absent
• Token-level Adaptive Training for Neural Machine Translation
• Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models
• Neural Machine Translation: A Review and Survey
• Semi-Autoregressive Image Captioning
• Designing Multimodal Datasets for NLP Challenges