Revisiting Self-Training for Neural Sequence Generation

Abstract

Self-training is one of the earliest and simplest semi-supervised methods. The key idea is to augment the original labeled dataset with unlabeled data paired with the model's prediction (i.e. the pseudo-parallel data). While self-training has been extensively studied on classification problems, in complex sequence generation tasks (e.g. machine translation) it is still unclear how self-training works due to the compositionality of the target space. In this work, we first empirically show that self-training is able to decently improve the supervised baseline on neural sequence generation tasks. Through careful examination of the performance gains, we find that the perturbation on the hidden states (i.e. dropout) is critical for self-training to benefit from the pseudo-parallel data, which acts as a regularizer and forces the model to yield close predictions for similar unlabeled inputs. Such effect helps the model correct some incorrect predictions on unlabeled data. To further encourage this mechanism, we propose to inject noise to the input space, resulting in a "noisy" version of self-training. Empirical study on standard machine translation and text summarization benchmarks shows that noisy self-training is able to effectively utilize unlabeled data and improve the performance of the supervised baseline by a large margin.

arXiv commentICLR 2020. The first two authors contributed equally. Updated to fix typos

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• Unsupervised Data Augmentation for Consistency Training
• Semi-Supervised Learning with Deep Generative Models
• Semi-Supervised Learning with Ladder Networks
• MASS: Masked Sequence to Sequence Pre-training for Language Generation

Cited by in corpus (21)

• Self-training with Noisy Student improves ImageNet classification
• Improved Noisy Student Training for Automatic Speech Recognition
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Margin Preserving Self-paced Contrastive Learning Towards Domain Adaptation for Medical Image Segmentation
• Self-training Improves Pre-training for Natural Language Understanding
• Unsupervised Pidgin Text Generation By Pivoting English Data and Self-Training
• Neural Machine Translation with Monolingual Translation Memory
• USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation
• Robust Mutual Learning for Semi-supervised Semantic Segmentation
• Magic dust for cross-lingual adaptation of monolingual wav2vec-2.0
• Noisy Self-Knowledge Distillation for Text Summarization
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation
• Facebook AI's WAT19 Myanmar-English Translation Task Submission
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data
• The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021
• Compositionality Through Language Transmission, using Artificial Neural Networks
• Self-Training with Weak Supervision
• Rumour Detection via Zero-shot Cross-lingual Transfer Learning
• Predictions For Pre-training Language Models
• Quantity vs. Quality of Monolingual Source Data in Automatic Text Translation: Can It Be Too Little If It Is Too Good?