SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models

Abstract

Sentence embedding is an important research topic in natural language processing (NLP) since it can transfer knowledge to downstream tasks. Meanwhile, a contextualized word representation, called BERT, achieves the state-of-the-art performance in quite a few NLP tasks. Yet, it is an open problem to generate a high quality sentence representation from BERT-based word models. It was shown in previous study that different layers of BERT capture different linguistic properties. This allows us to fusion information across layers to find better sentence representation. In this work, we study the layer-wise pattern of the word representation of deep contextualized models. Then, we propose a new sentence embedding method by dissecting BERT-based word models through geometric analysis of the space spanned by the word representation. It is called the SBERT-WK method. No further training is required in SBERT-WK. We evaluate SBERT-WK on semantic textual similarity and downstream supervised tasks. Furthermore, ten sentence-level probing tasks are presented for detailed linguistic analysis. Experiments show that SBERT-WK achieves the state-of-the-art performance. Our codes are publicly available.

References in corpus (10)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• SentEval: An Evaluation Toolkit for Universal Sentence Representations
• Evaluating Word Embedding Models: Methods and Experimental Results
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Language Models as Knowledge Bases?
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• Visualizing and Understanding the Effectiveness of BERT

Cited by in corpus (6)

• Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Multitask Learning for Class-Imbalanced Discourse Classification
• Efficient Sentence Embedding via Semantic Subspace Analysis
• Vec2Sent: Probing Sentence Embeddings with Natural Language Generation
• A New Sentence Ordering Method Using BERT Pretrained Model