Enhancing Sentence Embedding with Generalized Pooling

Abstract

Pooling is an essential component of a wide variety of sentence representation and embedding models. This paper explores generalized pooling methods to enhance sentence embedding. We propose vector-based multi-head attention that includes the widely used max pooling, mean pooling, and scalar self-attention as special cases. The model benefits from properly designed penalization terms to reduce redundancy in multi-head attention. We evaluate the proposed model on three different tasks: natural language inference (NLI), author profiling, and sentiment classification. The experiments show that the proposed model achieves significant improvement over strong sentence-encoding-based methods, resulting in state-of-the-art performances on four datasets. The proposed approach can be easily implemented for more problems than we discuss in this paper.

Accepted by COLING 2018

References in corpus (4)

• Distributed Representations of Sentences and Documents
• Skip-Thought Vectors
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering

Cited by in corpus (16)

• Attention in Natural Language Processing
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Multilingual NMT with a language-independent attention bridge
• Sequential Attention-based Network for Noetic End-to-End Response Selection
• Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification
• Conditional Self-Attention for Query-based Summarization
• Improving BERT with Self-Supervised Attention
• Several Experiments on Investigating Pretraining and Knowledge-Enhanced Models for Natural Language Inference
• Cell-aware Stacked LSTMs for Modeling Sentences
• TextNAS: A Neural Architecture Search Space tailored for Text Representation
• Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
• Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representations
• REGMAPR - Text Matching Made Easy
• Humanly Certifying Superhuman Classifiers
• On Training Instance Selection for Few-Shot Neural Text Generation
• Exploring End-to-End Differentiable Natural Logic Modeling