Semi-Supervised Sequence Modeling with Cross-View Training

Abstract

Unsupervised representation learning algorithms such as word2vec and ELMo improve the accuracy of many supervised NLP models, mainly because they can take advantage of large amounts of unlabeled text. However, the supervised models only learn from task-specific labeled data during the main training phase. We therefore propose Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data. On labeled examples, standard supervised learning is used. On unlabeled examples, CVT teaches auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) to match the predictions of the full model seeing the whole input. Since the auxiliary modules and the full model share intermediate representations, this in turn improves the full model. Moreover, we show that CVT is particularly effective when combined with multi-task learning. We evaluate CVT on five sequence tagging tasks, machine translation, and dependency parsing, achieving state-of-the-art results.

arXiv commentEMNLP 2018

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• mixup: Beyond Empirical Risk Minimization
• An Overview of Multi-Task Learning in Deep Neural Networks
• Improved Techniques for Training GANs
• A Survey on Multi-view Learning
• Effective Approaches to Attention-based Neural Machine Translation
• Born Again Neural Networks
• Learning with Pseudo-Ensembles
• Distributional Smoothing with Virtual Adversarial Training
• Introduction to the CoNLL-2000 Shared Task: Chunking
• Learned in Translation: Contextualized Word Vectors
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Neural Probabilistic Model for Non-projective MST Parsing
• Deep Multitask Learning for Semantic Dependency Parsing
• Shortcut Sequence Tagging

Cited by in corpus (12)

• Unsupervised Data Augmentation for Consistency Training
• Natural Language Processing Advancements By Deep Learning: A Survey
• A Survey of Deep Learning for Scientific Discovery
• Adaptive Self-training for Few-shot Neural Sequence Labeling
• MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
• Transformer to CNN: Label-scarce distillation for efficient text classification
• To Tune or Not To Tune? How About the Best of Both Worlds?
• Constructive Type-Logical Supertagging with Self-Attention Networks
• Supertagging Combinatory Categorial Grammar with Attentive Graph Convolutional Networks
• RPCL: A Framework for Improving Cross-Domain Detection with Auxiliary Tasks
• FlexNER: A Flexible LSTM-CNN Stack Framework for Named Entity Recognition
• Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering