Distance-based Self-Attention Network for Natural Language Inference

Abstract

Attention mechanism has been used as an ancillary means to help RNN or CNN. However, the Transformer (Vaswani et al., 2017) recently recorded the state-of-the-art performance in machine translation with a dramatic reduction in training time by solely using attention. Motivated by the Transformer, Directional Self Attention Network (Shen et al., 2017), a fully attention-based sentence encoder, was proposed. It showed good performance with various data by using forward and backward directional information in a sentence. But in their study, not considered at all was the distance between words, an important feature when learning the local dependency to help understand the context of input text. We propose Distance-based Self-Attention Network, which considers the word distance by using a simple distance mask in order to model the local dependency without losing the ability of modeling global dependency which attention has inherent. Our model shows good performance with NLI data, and it records the new state-of-the-art result with SNLI data. Additionally, we show that our model has a strength in long sentences or documents.

12 pages, 13 figures

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Character-Aware Neural Language Models
• Layer Normalization
• A large annotated corpus for learning natural language inference
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
• Dynamic Routing Between Capsules
• Order-Embeddings of Images and Language
• Natural Language Inference by Tree-Based Convolution and Heuristic Matching
• Shortcut-Stacked Sentence Encoders for Multi-Domain Inference
• Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference

Cited by in corpus (18)

• The Natural Language Decathlon: Multitask Learning as Question Answering
• Dynamic Self-Attention : Computing Attention over Words Dynamically for Sentence Embedding
• Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
• Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding
• Towards Open Intent Discovery for Conversational Text
• LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Attention Boosted Sequential Inference Model
• SNU_IDS at SemEval-2019 Task 3: Addressing Training-Test Class Distribution Mismatch in Conversational Classification
• Integrated Eojeol Embedding for Erroneous Sentence Classification in Korean Chatbots
• Neural Machine Translation: A Review and Survey
• REGMAPR - Text Matching Made Easy
• R${}^2$-Net: Relation of Relation Learning Network for Sentence Semantic Matching
• Multiple Structural Priors Guided Self Attention Network for Language Understanding
• Highway Transformer: Self-Gating Enhanced Self-Attentive Networks
• DGA-Net Dynamic Gaussian Attention Network for Sentence Semantic Matching
• Modeling Concentrated Cross-Attention for Neural Machine Translation with Gaussian Mixture Model