DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Abstract

Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)", is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.

arXiv comment10 pages, 8 figures; Accepted in AAAI-18

References in corpus (11)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• ADADELTA: An Adaptive Learning Rate Method
• Character-Aware Neural Language Models
• Effective Approaches to Attention-based Neural Machine Translation
• A large annotated corpus for learning natural language inference
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• Neural Responding Machine for Short-Text Conversation
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Reinforced Mnemonic Reader for Machine Reading Comprehension
• Bidirectional Tree-Structured LSTM with Head Lexicalization
• Learning Structured Text Representations

Cited by in corpus (30)

• Attention U-Net: Learning Where to Look for the Pancreas
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Modality specific U-Net variants for biomedical image segmentation: A survey
• Semi-supervised Credit Card Fraud Detection via Attribute-Driven Graph Representation
• Neural Network Models for Paraphrase Identification, Semantic Textual Similarity, Natural Language Inference, and Question Answering
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation
• Distance-based Self-Attention Network for Natural Language Inference
• An Introductory Survey on Attention Mechanisms in NLP Problems
• Joint Slot Filling and Intent Detection via Capsule Neural Networks
• Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling
• Reasoning with Sarcasm by Reading In-between
• Conditional Self-Attention for Query-based Summarization
• Pop Music Highlighter: Marking the Emotion Keypoints
• Unsupervised Neural Machine Translation with Weight Sharing
• Hybrid Self-Attention Network for Machine Translation
• Multi-turn Inference Matching Network for Natural Language Inference
• Exploiting Coarse-to-Fine Task Transfer for Aspect-level Sentiment Classification
• DAGCN: Dual Attention Graph Convolutional Networks
• Building Sequential Inference Models for End-to-End Response Selection
• Multi-scale Attention U-Net (MsAUNet): A Modified U-Net Architecture for Scene Segmentation
• Improving Neural Machine Translation with Pre-trained Representation
• FineText: Text Classification via Attention-based Language Model Fine-tuning
• Self-Attentive Neural Collaborative Filtering
• Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations
• Linguistically-Enriched and Context-Aware Zero-shot Slot Filling
• Self-Attention-Based Message-Relevant Response Generation for Neural Conversation Model
• AAANE: Attention-based Adversarial Autoencoder for Multi-scale Network Embedding
• DGA-Net Dynamic Gaussian Attention Network for Sentence Semantic Matching
• CASA-NLU: Context-Aware Self-Attentive Natural Language Understanding for Task-Oriented Chatbots