Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures

Abstract

Measuring sentence semantic similarity using pre-trained language models such as BERT generally yields unsatisfactory zero-shot performance, and one main reason is ineffective token aggregation methods such as mean pooling. In this paper, we demonstrate under a Bayesian framework that distance between primitive statistics such as the mean of word embeddings are fundamentally flawed for capturing sentence-level semantic similarity. To remedy this issue, we propose to learn a categorical variational autoencoder (VAE) based on off-the-shelf pre-trained language models. We theoretically prove that measuring the distance between the latent categorical mixtures, namely Latte-Mix, can better reflect the true sentence semantic similarity. In addition, our Bayesian framework provides explanations for why models finetuned on labelled sentence pairs have better zero-shot performance. We also empirically demonstrate that these finetuned models could be further improved by Latte-Mix. Our method not only yields the state-of-the-art zero-shot performance on semantic similarity datasets such as STS, but also enjoy the benefits of fast training and having small memory footprints.

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Distributed Representations of Sentences and Documents
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Universal Sentence Encoder
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Skip-Thought Vectors
• A large annotated corpus for learning natural language inference
• An efficient framework for learning sentence representations
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• A Mutual Information Maximization Perspective of Language Representation Learning
• How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• Probabilistic FastText for Multi-Sense Word Embeddings
• Embedding Words as Distributions with a Bayesian Skip-gram Model
• Learning Generic Sentence Representations Using Convolutional Neural Networks