Semi-Supervised Text Classification via Self-Pretraining

Abstract

We present a neural semi-supervised learning model termed Self-Pretraining. Our model is inspired by the classic self-training algorithm. However, as opposed to self-training, Self-Pretraining is threshold-free, it can potentially update its belief about previously labeled documents, and can cope with the semantic drift problem. Self-Pretraining is iterative and consists of two classifiers. In each iteration, one classifier draws a random set of unlabeled documents and labels them. This set is used to initialize the second classifier, to be further trained by the set of labeled documents. The algorithm proceeds to the next iteration and the classifiers' roles are reversed. To improve the flow of information across the iterations and also to cope with the semantic drift problem, Self-Pretraining employs an iterative distillation process, transfers hypotheses across the iterations, utilizes a two-stage training model, uses an efficient learning rate schedule, and employs a pseudo-label transformation heuristic. We have evaluated our model in three publicly available social media datasets. Our experiments show that Self-Pretraining outperforms the existing state-of-the-art semi-supervised classifiers across multiple settings. Our code is available at https://github.com/p-karisani/self_pretraining.

arXiv commentWSDM 2021

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Simple Framework for Contrastive Learning of Visual Representations
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Language Models are Few-Shot Learners
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Asymmetric Tri-training for Unsupervised Domain Adaptation
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Uncertainty-aware Self-training for Text Classification with Few Labels
• A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering