A Structured Self-attentive Sentence Embedding

Abstract

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

15 pages with appendix, 7 figures, 4 tables. Conference paper in 5th International Conference on Learning Representations (ICLR 2017)

Cited by in corpus (6)

• Weighted Transformer Network for Machine Translation
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models
• Clickbait Detection in Tweets Using Self-attentive Network
• Highrisk Prediction from Electronic Medical Records via Deep Attention Networks
• Improving Visually Grounded Sentence Representations with Self-Attention
• Testing the limits of unsupervised learning for semantic similarity