Adversarial Training Methods for Semi-Supervised Text Classification

Abstract

Adversarial training provides a means of regularizing supervised learning algorithms while virtual adversarial training is able to extend supervised learning algorithms to the semi-supervised setting. However, both methods require making small perturbations to numerous entries of the input vector, which is inappropriate for sparse high-dimensional inputs such as one-hot word representations. We extend adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself. The proposed method achieves state of the art results on multiple benchmark semi-supervised and purely supervised tasks. We provide visualizations and analysis showing that the learned word embeddings have improved in quality and that while training, the model is less prone to overfitting. Code is available at https://github.com/tensorflow/models/tree/master/research/adversarial_text.

Published as a conference paper at ICLR 2017

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Distributional Smoothing with Virtual Adversarial Training
• Auxiliary Deep Generative Models
• Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
• Analyzing noise in autoencoders and deep networks
• Self-Adaptive Hierarchical Sentence Model
• Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level

Cited by in corpus (68)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Unsupervised Data Augmentation for Consistency Training
• A Survey on Data Augmentation for Text Classification
• Adversarial Personalized Ranking for Recommendation
• Deep Learning Based Text Classification: A Comprehensive Review
• Learning to Generate Reviews and Discovering Sentiment
• TextBugger: Generating Adversarial Text Against Real-world Applications
• Quasi-Recurrent Neural Networks
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Pathologies of Neural Models Make Interpretations Difficult
• DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars
• Weakly-Supervised Neural Text Classification
• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
• Feature-map-level Online Adversarial Knowledge Distillation
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function
• Adversarial Deep Structural Networks for Mammographic Mass Segmentation
• Technical report on Conversational Question Answering
• Learning perturbation sets for robust machine learning
• A Neural Entity Coreference Resolution Review
• Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level
• Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification
• Improving adversarial robustness of deep neural networks by using semantic information
• MixUp as Directional Adversarial Training
• Achieving Adversarial Robustness via Sparsity
• A Way out of the Odyssey: Analyzing and Combining Recent Insights for LSTMs
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Towards Open Intent Discovery for Conversational Text
• Gradient-Based Adversarial Training on Transformer Networks for Detecting Check-Worthy Factual Claims
• Adversarial Training with Contrastive Learning in NLP
• Learning to Impute: A General Framework for Semi-supervised Learning
• Image Quality Assessment Techniques Show Improved Training and Evaluation of Autoencoder Generative Adversarial Networks
• Manifold Adversarial Learning
• Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection
• kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
• Advancing PICO Element Detection in Biomedical Text via Deep Neural Networks
• T3: Tree-Autoencoder Constrained Adversarial Text Generation for Targeted Attack
• GRACE: Gradient Harmonized and Cascaded Labeling for Aspect-based Sentiment Analysis
• How Does Adversarial Fine-Tuning Benefit BERT?
• ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Data
• Ptolemy: Architecture Support for Robust Deep Learning
• Modeling EEG data distribution with a Wasserstein Generative Adversarial Network to predict RSVP Events
• Virtual Adversarial Training on Graph Convolutional Networks in Node Classification
• Adversarial Transformations for Semi-Supervised Learning
• Understanding and Improving Virtual Adversarial Training
• Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests
• FineFool: Fine Object Contour Attack via Attention
• Addressing the Vulnerability of NMT in Input Perturbations
• An Adversarially-Learned Turing Test for Dialog Generation Models
• Adversarial Dropout for Recurrent Neural Networks
• Search Space of Adversarial Perturbations against Image Filters
• Learning low dimensional word based linear classifiers using Data Shared Adaptive Bootstrap Aggregated Lasso with application to IMDb data
• Adversarial Learning for Supervised and Semi-supervised Relation Extraction in Biomedical Literature
• User-Guided Aspect Classification for Domain-Specific Texts
• Towards Controlled Transformation of Sentiment in Sentences
• Multiple Document Representations from News Alerts for Automated Bio-surveillance Event Detection
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification
• Orthogonal Deep Models As Defense Against Black-Box Attacks
• Adversarial Ladder Networks
• Uncertainty-Aware Data Aggregation for Deep Imitation Learning
• Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweets
• On Robustness of Neural Semantic Parsers
• Metric Learning for Dynamic Text Classification
• A Diversity-Enhanced and Constraints-Relaxed Augmentation for Low-Resource Classification
• Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training
• Learn to Resolve Conversational Dependency: A Consistency Training Framework for Conversational Question Answering
• Generalizing Neural Networks by Reflecting Deviating Data in Production