Extracting Sentence Embeddings from Pretrained Transformer Models

Abstract

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT's hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT's sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

Postprint update

References in corpus (16)

• Sequence to Sequence Learning with Neural Networks
• Explaining and Harnessing Adversarial Examples
• Distributed Representations of Sentences and Documents
• SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Self-Taught Convolutional Neural Networks for Short Text Clustering
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Efficient Vector Representation for Documents through Corruption
• Multi-Step Regression Learning for Compositional Distributional Semantics
• Pair-Level Supervised Contrastive Learning for Natural Language Inference
• Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors
• IsoScore: Measuring the Uniformity of Embedding Space Utilization
• Language Through a Prism: A Spectral Approach for Multiscale Language Representations
• Product Classification in E-Commerce using Distributional Semantics
• Improving Document Classification with Multi-Sense Embeddings