Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding

Abstract

In this paper, we propose Dynamic Self-Attention (DSA), a new self-attention mechanism for sentence embedding. We design DSA by modifying dynamic routing in capsule network (Sabouretal.,2017) for natural language processing. DSA attends to informative words with a dynamic weight vector. We achieve new state-of-the-art results among sentence encoding methods in Stanford Natural Language Inference (SNLI) dataset with the least number of parameters, while showing comparative results in Stanford Sentiment Treebank (SST) dataset.

arXiv comment7 pages, 4 figures

References in corpus (5)

• Convolutional Sequence to Sequence Learning
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Distance-based Self-Attention Network for Natural Language Inference
• Shortcut-Stacked Sentence Encoders for Multi-Domain Inference

Cited by in corpus (12)

• Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering
• LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching
• Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference
• Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logic
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Multi-Scale Self-Attention for Text Classification
• Integrated Eojeol Embedding for Erroneous Sentence Classification in Korean Chatbots
• Exploring End-to-End Differentiable Natural Logic Modeling
• DGA-Net Dynamic Gaussian Attention Network for Sentence Semantic Matching
• R${}^2$-Net: Relation of Relation Learning Network for Sentence Semantic Matching
• Multiple Structural Priors Guided Self Attention Network for Language Understanding
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching