Extracting Sentence Embeddings from Pretrained Transformer Models

Abstract

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT's hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT's sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

Postprint update

References in corpus (35)

• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Explaining and Harnessing Adversarial Examples
• Distributed Representations of Sentences and Documents
• Estimating Mutual Information
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Self-Taught Convolutional Neural Networks for Short Text Clustering
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• What do you learn from context? Probing for sentence structure in contextualized word representations
• All-but-the-Top: Simple and Effective Postprocessing for Word Representations
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Efficient Vector Representation for Documents through Corruption
• Multi-Step Regression Learning for Compositional Distributional Semantics
• Representation Degeneration Problem in Training Natural Language Generation Models
• Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
• Pair-Level Supervised Contrastive Learning for Natural Language Inference
• Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors
• IsoScore: Measuring the Uniformity of Embedding Space Utilization
• Language Through a Prism: A Spectral Approach for Multiscale Language Representations
• Rethinking embedding coupling in pre-trained language models
• Product Classification in E-Commerce using Distributional Semantics
• Improving Document Classification with Multi-Sense Embeddings
• Testing pre-trained Transformer models for Lithuanian news clustering
• Words are not Equal: Graded Weighting Model for building Composite Document Vectors
• Evaluation of Unsupervised Compositional Representations
• Unsupervised Post-processing of Word Vectors via Conceptor Negation
• Combining Axiom Injection and Knowledge Base Completion for Efficient Natural Language Inference