Perturbation-based Self-supervised Attention for Attention Bias in Text Classification

Abstract

In text classification, the traditional attention mechanisms usually focus too much on frequent words, and need extensive labeled data in order to learn. This paper proposes a perturbation-based self-supervised attention approach to guide attention learning without any annotation overhead. Specifically, we add as much noise as possible to all the words in the sentence without changing their semantics and predictions. We hypothesize that words that tolerate more noise are less significant, and we can use this information to refine the attention distribution. Experimental results on three text classification tasks show that our approach can significantly improve the performance of current attention-based models, and is more effective than existing self-supervised methods. We also provide a visualization analysis to verify the effectiveness of our approach.

References in corpus (12)

• Neural Machine Translation by Jointly Learning to Align and Translate
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Structured Self-attentive Sentence Embedding
• Enhanced LSTM for Natural Language Inference
• Attention is not Explanation
• Understanding Neural Networks through Representation Erasure
• A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees
• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs
• Assessing Robustness of Text Classification through Maximal Safe Radius Computation
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification