Self-supervised Regularization for Text Classification

Abstract

Text classification is a widely studied problem and has broad applications. In many real-world problems, the number of texts for training classification models is limited, which renders these models prone to overfitting. To address this problem, we propose SSL-Reg, a data-dependent regularization approach based on self-supervised learning (SSL). SSL is an unsupervised learning approach which defines auxiliary tasks on input data without using any human-provided labels and learns data representations by solving these auxiliary tasks. In SSL-Reg, a supervised classification task and an unsupervised SSL task are performed simultaneously. The SSL task is unsupervised, which is defined purely on input texts without using any human-provided labels. Training a model using an SSL task can prevent the model from being overfitted to a limited number of class labels in the classification task. Experiments on 17 text classification datasets demonstrate the effectiveness of our proposed method.

arXiv comment16 pages, 3 figures, to be published in Transactions of the Association for Computational Linguistics

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Framework for Contrastive Learning of Visual Representations
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Momentum Contrast for Unsupervised Visual Representation Learning
• A C-LSTM Neural Network for Text Classification
• Deep Learning Based Text Classification: A Comprehensive Review
• Improvements to context based self-supervised learning
• Supervised Contrastive Learning
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• Joint-task Self-supervised Learning for Temporal Correspondence
• Contrastive Self-supervised Learning for Graph Classification
• Importance-Aware Learning for Neural Headline Editing