Enhancing Sentence Embedding with Generalized Pooling
arXiv:1806.09828
Abstract
Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.
arXiv commentAccepted by COLING 2018
References in corpus (8)
- Efficient Estimation of Word Representations in Vector Space
- Distributed Representations of Sentences and Documents
- Enhanced LSTM for Natural Language Inference
- Skip-Thought Vectors
- Reasoning about Entailment with Neural Attention
- Neural Natural Language Inference Models Enhanced with External Knowledge
- Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
- Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
Cited by in corpus (16)
- Attention in Natural Language Processing
- COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
- Multilingual NMT with a language-independent attention bridge
- Sequential Attention-based Network for Noetic End-to-End Response Selection
- Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification
- Conditional Self-Attention for Query-based Summarization
- Improving BERT with Self-Supervised Attention
- Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference
- Cell-aware Stacked LSTMs for Modeling Sentences
- TextNAS: A Neural Architecture Search Space tailored for Text Representation
- Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
- Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations
- REGMAPR - Text Matching Made Easy
- Humanly Certifying Superhuman Classifiers
- On Training Instance Selection for Few-Shot Neural Text Generation
- Exploring End-to-End Differentiable Natural Logic Modeling