A Structured Self-attentive Sentence Embedding

Abstract

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification, and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

15 pages with appendix, 7 figures, 4 tables. Conference paper in 5th International Conference on Learning Representations (ICLR 2017)

Cited by in corpus (326)

• Attention Mechanisms in Computer Vision: A Survey
• DASNet: Dual attentive fully convolutional siamese networks for change detection of high resolution satellite images
• A General Survey on Attention Mechanisms in Deep Learning
• Attention-based Deep Multiple Instance Learning
• Attention in Natural Language Processing
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks
• OCNet: Object Context Network for Scene Parsing
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Representation Learning for Attributed Multiplex Heterogeneous Network
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• Deep Learning Based Text Classification: A Comprehensive Review
• Actor-Attention-Critic for Multi-Agent Reinforcement Learning
• Neural Natural Language Inference Models Enhanced with External Knowledge
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Neural Relational Inference for Interacting Systems
• Object-Centric Learning with Slot Attention
• Dual Attention Network for Scene Segmentation
• Adversarial Text-to-Image Synthesis: A Review
• Deep ConvLSTM with self-attention for human activity decoding using wearables
• MHSA-Net: Multi-Head Self-Attention Network for Occluded Person Re-Identification
• RMDL: Recalibrated multi-instance deep learning for whole slide gastric image classification
• Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record
• Language Modeling with Deep Transformers
• ST-GRAT: A Novel Spatio-temporal Graph Attention Network for Accurately Forecasting Dynamically Changing Road Speed
• A Label Attention Model for ICD Coding from Clinical Text
• Few-Shot Graph Learning for Molecular Property Prediction
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• Causality Extraction based on Self-Attentive BiLSTM-CRF with Transferred Embeddings
• Weighted Transformer Network for Machine Translation
• A Hierarchical Structured Self-Attentive Model for Extractive Document Summarization (HSSAS)
• Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networks
• Recent Progress in Transformer-based Medical Image Analysis
• A Survey on Text Classification: From Shallow to Deep Learning
• How to Fine-Tune BERT for Text Classification?
• Deep Enhanced Representation for Implicit Discourse Relation Recognition
• Weakly Labelled AudioSet Tagging with Attention Neural Networks
• Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions
• Explainable AI: current status and future directions
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Dive into Deep Learning
• FaceFilter: Audio-visual speech separation using still images
• Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models
• Amortized Causal Discovery: Learning to Infer Causal Graphs from Time-Series Data
• Attention-guided Context Feature Pyramid Network for Object Detection
• Emotion recognition by fusing time synchronous and time asynchronous representations
• Multi-Scale Feature Fusion: Learning Better Semantic Segmentation for Road Pothole Detection
• Multiple-Human Parsing in the Wild
• Towards Accurate Scene Text Recognition with Semantic Reasoning Networks
• A Universal Representation Transformer Layer for Few-Shot Image Classification
• Few-shot Text Classification with Distributional Signatures
• Adaptive Online Incremental Learning for Evolving Data Streams
• Is Attention Better Than Matrix Decomposition?
• FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• CRaDLe: Deep Code Retrieval Based on Semantic Dependency Learning
• Exploring the State of the Art in Legal QA Systems
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
• Clickbait Detection in Tweets Using Self-attentive Network
• A Survey on State-of-the-art Deep Learning Applications and Challenges
• Multi-layer Representation Fusion for Neural Machine Translation
• Real-time Attention Based Look-alike Model for Recommender System
• Human Activity Recognition from Wearable Sensor Data Using Self-Attention
• Describing a Knowledge Base
• End-to-End Neural Diarization: Reformulating Speaker Diarization as Simple Multi-label Classification
• Multi-level Contrastive Learning Framework for Sequential Recommendation
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain
• A New Training Pipeline for an Improved Neural Transducer
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• Deep Learning Based Chatbot Models
• Temporal Self-Attention Network for Medical Concept Embedding
• Toward Interpretable Music Tagging with Self-Attention
• Multilingual NMT with a language-independent attention bridge
• DropAttention: A Regularization Method for Fully-Connected Self-Attention Networks
• Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention
• Cycle-Contrast for Self-Supervised Video Representation Learning
• An AutoML-based Approach to Multimodal Image Sentiment Analysis
• Position-Aware Self-Attention based Neural Sequence Labeling
• Constraint Translation Candidates: A Bridge between Neural Query Translation and Cross-lingual Information Retrieval
• Sequential Attention-based Network for Noetic End-to-End Response Selection
• Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training
• Data-Driven Methods for Solving Algebra Word Problems
• Signed Distance-based Deep Memory Recommender
• BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition
• Exploiting Neural Query Translation into Cross Lingual Information Retrieval
• Context-Aware Self-Attention Networks
• Knowledge Graphs and Pre-trained Language Models enhanced Representation Learning for Conversational Recommender Systems
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Attention Back-end for Automatic Speaker Verification with Multiple Enrollment Utterances
• Towards Knowledge-Based Recommender Dialog System
• Can We Generate Shellcodes via Natural Language? An Empirical Study
• Climate and Weather: Inspecting Depression Detection via Emotion Recognition
• Highrisk Prediction from Electronic Medical Records via Deep Attention Networks
• Attention-Based Neural Networks for Chroma Intra Prediction in Video Coding
• BERT-like Pre-training for Symbolic Piano Music Classification Tasks
• Convolutional Self-Attention Networks
• Contrastive Attention for Automatic Chest X-ray Report Generation
• Stochastic Answer Networks for SQuAD 2.0
• Combination of Deep Speaker Embeddings for Diarisation
• An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
• Self-Attentive Residual Decoder for Neural Machine Translation
• Improving One-stage Visual Grounding by Recursive Sub-query Construction
• Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
• A Study of Reinforcement Learning for Neural Machine Translation
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• Controllable Multi-Interest Framework for Recommendation
• Saliency-Guided Attention Network for Image-Sentence Matching
• Induction Networks for Few-Shot Text Classification
• Attention-Based Capsule Networks with Dynamic Routing for Relation Extraction
• Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog
• Learning Generic Sentence Representations Using Convolutional Neural Networks
• Zero-shot User Intent Detection via Capsule Neural Networks
• AliGraph: A Comprehensive Graph Neural Network Platform
• Distantly Supervised Relation Extraction via Recursive Hierarchy-Interactive Attention and Entity-Order Perception
• Spatial-Temporal Dynamic Graph Attention Networks for Ride-hailing Demand Prediction
• Distance-Free Modeling of Multi-Predicate Interactions in End-to-End Japanese Predicate-Argument Structure Analysis
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Information Aggregation for Multi-Head Attention with Routing-by-Agreement
• Keyphrase Extraction with Span-based Feature Representations
• Knowledge-aware Attention Network for Protein-Protein Interaction Extraction
• On the Importance of Attention in Meta-Learning for Few-Shot Text Classification
• End-to-End Neural Speaker Diarization with Self-attention
• Hierarchical Opacity Propagation for Image Matting
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• Neural Cross-Lingual Named Entity Recognition with Minimal Resources
• The Gap of Semantic Parsing: A Survey on Automatic Math Word Problem Solvers
• Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems
• CoVA: Context-aware Visual Attention for Webpage Information Extraction
• Code-switched inspired losses for generic spoken dialog representations
• TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition
• When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification
• Incorporating Heterogeneous User Behaviors and Social Influences for Predictive Analysis
• Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems
• Dynamically Pruned Message Passing Networks for Large-Scale Knowledge Graph Reasoning
• Online Deep Learning based on Auto-Encoder
• Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
• Sparse Attentive Memory Network for Click-through Rate Prediction with Long Sequences
• Attention-based Multi-instance Neural Network for Medical Diagnosis from Incomplete and Low Quality Data
• Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval
• CAN: Constrained Attention Networks for Multi-Aspect Sentiment Analysis
• Self-Attentive Document Interaction Networks for Permutation Equivariant Ranking
• FREA-Unet: Frequency-aware U-net for Modality Transfer
• A Time Attention based Fraud Transaction Detection Framework
• Enhancing Machine Translation with Dependency-Aware Self-Attention
• Towards Open Intent Discovery for Conversational Text
• Relational Graph Learning for Crowd Navigation
• RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models
• Unsupervised Features Extraction for Binary Similarity Using Graph Embedding Neural Networks
• Improving BERT with Self-Supervised Attention
• Multimodal deep learning for short-term stock volatility prediction
• Cross-Modal Retrieval with Implicit Concept Association
• Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
• Self-Attention with Structural Position Representations
• DAGCN: Dual Attention Graph Convolutional Networks
• Learning to score the figure skating sports videos
• Explaining The Efficacy of Counterfactually Augmented Data
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction
• Investigating an Effective Character-level Embedding in Korean Sentence Classification
• Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks
• Improving Visually Grounded Sentence Representations with Self-Attention
• Hybrid Attention-Based Transformer Block Model for Distant Supervision Relation Extraction
• Domain-Specific Priors and Meta Learning for Few-Shot First-Person Action Recognition
• Dialog State Tracking with Reinforced Data Augmentation
• Long Short-Term Attention
• Short Text Classification via Knowledge powered Attention with Similarity Matrix based CNN
• Neighbourhood-Insensitive Point Cloud Normal Estimation Network
• Generating Responses Expressing Emotion in an Open-domain Dialogue System
• Human Motion Analysis with Deep Metric Learning
• Modeling Graph Structure in Transformer for Better AMR-to-Text Generation
• Basic and Depression Specific Emotion Identification in Tweets: Multi-label Classification Experiments
• DEAL: Difficulty-aware Active Learning for Semantic Segmentation
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• Revisiting Classification Perspective on Scene Text Recognition
• Untangling tradeoffs between recurrence and self-attention in neural networks
• Notes on Deep Learning for NLP
• On Defining Smart Cities using Transformer Neural Networks
• ANA at SemEval-2019 Task 3: Contextual Emotion detection in Conversations through hierarchical LSTMs and BERT
• A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
• Self-Attention Enhanced Patient Journey Understanding in Healthcare System
• A Simple Yet Effective Method for Video Temporal Grounding with Cross-Modality Attention
• Learning Credible Deep Neural Networks with Rationale Regularization
• Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
• Gated Group Self-Attention for Answer Selection
• Sentiment Analysis with Contextual Embeddings and Self-Attention
• Tensorized Self-Attention: Efficiently Modeling Pairwise and Global Dependencies Together
• Modeling Document Interactions for Learning to Rank with Regularized Self-Attention
• Perturbation-based Self-supervised Attention for Attention Bias in Text Classification
• UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
• Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking
• DENS: A Dataset for Multi-class Emotion Analysis
• Variational Self-attention Model for Sentence Representation
• SACT: Self-Aware Multi-Space Feature Composition Transformer for Multinomial Attention for Video Captioning
• Learning Context-Sensitive Convolutional Filters for Text Processing
• Context-Aware Domain Adaptation in Semantic Segmentation
• Advancing PICO Element Detection in Biomedical Text via Deep Neural Networks
• Understanding Semantics from Speech Through Pre-training
• Spoofing-Aware Attention based ASV Back-end with Multiple Enrollment Utterances and a Sampling Strategy for the SASV Challenge 2022
• Deep Magnification-Flexible Upsampling over 3D Point Clouds
• Unsupervised Summarization by Jointly Extracting Sentences and Keywords
• Target Guided Emotion Aware Chat Machine
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• Attention-Guided Generative Adversarial Network to Address Atypical Anatomy in Modality Transfer
• Decision Support for Intoxication Prediction Using Graph Convolutional Networks
• Bayesian Transformer Language Models for Speech Recognition
• Investigation of learning abilities on linguistic features in sequence-to-sequence text-to-speech synthesis
• Self-Supervised Euphemism Detection and Identification for Content Moderation
• Learning to Ask Screening Questions for Job Postings
• NSURL-2019 Shared Task 8: Semantic Question Similarity in Arabic
• Combine Convolution with Recurrent Networks for Text Classification
• Watch It Twice: Video Captioning with a Refocused Video Encoder
• A Novel User Representation Paradigm for Making Personalized Candidate Retrieval
• Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models
• Text Classification for Azerbaijani Language Using Machine Learning and Embedding
• Predicting Diabetes Disease Evolution Using Financial Records and Recurrent Neural Networks
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
• Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language
• Inducing Grammars with and for Neural Machine Translation
• An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese
• Interpretable ICD Code Embeddings with Self- and Mutual-Attention Mechanisms
• Neural Text Classification by Jointly Learning to Cluster and Align
• Sequential Diagnosis Prediction with Transformer and Ontological Representation
• High-Order Paired-ASPP Networks for Semantic Segmenation
• AMAD: Adversarial Multiscale Anomaly Detection on High-Dimensional and Time-Evolving Categorical Data
• Text Matters but Speech Influences: A Computational Analysis of Syntactic Ambiguity Resolution
• Relevance-Promoting Language Model for Short-Text Conversation
• Speech Intention Understanding in a Head-final Language: A Disambiguation Utilizing Intonation-dependency
• Towards Better Modeling Hierarchical Structure for Self-Attention with Ordered Neurons
• Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER
• Interactive Classification by Asking Informative Questions
• A Multi-Size Neural Network with Attention Mechanism for Answer Selection
• Self-Attention with Cross-Lingual Position Representation
• Neural Machine Translation: A Review and Survey
• Improving Irregularly Sampled Time Series Learning with Dense Descriptors of Time
• Discriminative Clustering for Robust Unsupervised Domain Adaptation
• Low Rank Factorization for Compact Multi-Head Self-Attention
• A Weakly-Supervised Attention-based Visualization Tool for Assessing Political Affiliation
• Learning sentence embeddings using Recursive Networks
• Transformer for Emotion Recognition
• Learning Class-Transductive Intent Representations for Zero-shot Intent Detection
• EmotionX-DLC: Self-Attentive BiLSTM for Detecting Sequential Emotions in Dialogue
• Content-Aware Speaker Embeddings for Speaker Diarisation
• Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
• Relation Module for Non-answerable Prediction on Question Answering
• A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act Detection
• Reliable uncertainty estimate for antibiotic resistance classification with Stochastic Gradient Langevin Dynamics
• Shift-and-Balance Attention
• Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections
• MIRNet: Learning multiple identities representations in overlapped speech
• Feature boosting with efficient attention for scene parsing
• End-to-End Trainable Self-Attentive Shallow Network for Text-Independent Speaker Verification
• Graph-to-Sequence Neural Machine Translation
• Multi-task Sentence Encoding Model for Semantic Retrieval in Question Answering Systems
• Constructing Explainable Opinion Graphs from Review
• Picking Apart Story Salads
• Multi-Granular Text Encoding for Self-Explaining Categorization
• Content-based feature exploration for transparent music recommendation using self-attentive genre classification
• Orthonormal Embedding-based Deep Clustering for Single-channel Speech Separation
• RDPD: Rich Data Helps Poor Data via Imitation
• A Domain Generalization Perspective on Listwise Context Modeling
• On Tree-Based Neural Sentence Modeling
• You Only Look & Listen Once: Towards Fast and Accurate Visual Grounding
• Inducing Alignment Structure with Gated Graph Attention Networks for Sentence Matching
• Cross-Thought for Sentence Encoder Pre-training
• A Multi-sentiment-resource Enhanced Attention Network for Sentiment Classification
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• Self-attention Multi-view Representation Learning with Diversity-promoting Complementarity
• Discriminative Cross-View Binary Representation Learning
• Text classification with pixel embedding
• Neural Consciousness Flow
• Toward Interpretability of Dual-Encoder Models for Dialogue Response Suggestions
• Detect and Classify -- Joint Span Detection and Classification for Health Outcomes
• Incremental Learning from Scratch for Task-Oriented Dialogue Systems
• Enriching Article Recommendation with Phrase Awareness
• Deep Domain Adaptation under Deep Label Scarcity
• CoRGi: Content-Rich Graph Neural Networks with Attention
• Attentive Semantic Role Labeling with Boundary Indicator
• Pseudo Labeling and Negative Feedback Learning for Large-scale Multi-label Domain Classification
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers
• Sparse-Interest Network for Sequential Recommendation
• Dementia Severity Classification under Small Sample Size and Weak Supervision in Thick Slice MRI
• Multi-Faceted Representation Learning with Hybrid Architecture for Time Series Classification
• DCA: Diversified Co-Attention towards Informative Live Video Commenting
• MCSAE: Masked Cross Self-Attentive Encoding for Speaker Embedding
• Inverse Graph Identification: Can We Identify Node Labels Given Graph Labels?
• Learning to Compose over Tree Structures via POS Tags
• Hierarchical Attention: What Really Counts in Various NLP Tasks
• Cross-Level Cross-Scale Cross-Attention Network for Point Cloud Representation
• Careful analysis of XRD patterns with Attention
• Structured Self-Attention Weights Encode Semantics in Sentiment Analysis
• Comparative Visual Analytics for Assessing Medical Records with Sequence Embedding
• An In-depth Walkthrough on Evolution of Neural Machine Translation
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes
• Corpus-level and Concept-based Explanations for Interpretable Document Classification
• Grounding Complex Navigational Instructions Using Scene Graphs
• Relation Extraction from Biomedical and Clinical Text: Unified Multitask Learning Framework
• Testing the limits of unsupervised learning for semantic similarity
• Unsupervised Learning of Parsimonious General-Purpose Embeddings for User and Location Modelling
• To Understand Representation of Layer-aware Sequence Encoders as Multi-order-graph
• Text Information Aggregation with Centrality Attention
• Hybrid Encoder: Towards Efficient and Precise Native AdsRecommendation via Hybrid Transformer Encoding Networks
• Summarizing Videos with Attention
• Causal Discovery from Conditionally Stationary Time Series
• TB-Net: A Three-Stream Boundary-Aware Network for Fine-Grained Pavement Disease Segmentation
• Exploiting Sentence Embedding for Medical Question Answering
• Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
• CNN-based Dual-Chain Models for Knowledge Graph Learning
• Low-Resource Named Entity Recognition Based on Multi-hop Dependency Trigger
• SAFE: Self-Attentive Function Embeddings for Binary Similarity
• SAVEHR: Self Attention Vector Representations for EHR based Personalized Chronic Disease Onset Prediction and Interpretability
• Sequential Learning of Convolutional Features for Effective Text Classification
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
• Game-Based Video-Context Dialogue
• Developing neural machine translation models for Hungarian-English
• Evaluating Biomedical BERT Models for Vocabulary Alignment at Scale in the UMLS Metathesaurus
• Interpretable Filter Learning Using Soft Self-attention For Raw Waveform Speech Recognition
• Text Classification with Lexicon from PreAttention Mechanism