Towards better substitution-based word sense induction

Abstract

Word sense induction (WSI) is the task of unsupervised clustering of word usages within a sentence to distinguish senses. Recent work obtain strong results by clustering lexical substitutes derived from pre-trained RNN language models (ELMo). Adapting the method to BERT improves the scores even further. We extend the previous method to support a dynamic rather than a fixed number of clusters as supported by other prominent methods, and propose a method for interpreting the resulting clusters by associating them with their most informative substitutes. We then perform extensive error analysis revealing the remaining sources of errors in the WSI task. Our code is available at https://github.com/asafamr/bertwsi.

5 pages, 1 figure

References in corpus (1)

• Semi-supervised Word Sense Disambiguation with Neural Models

Cited by in corpus (10)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Capturing Evolution in Word Usage: Just Add More Clusters?
• Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution
• Rethinking embedding coupling in pre-trained language models
• BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection
• Characterizing English Variation across Social Media Communities with BERT
• Word Sense Induction with Hierarchical Clustering and Mutual Information Maximization
• Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
• To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models
• Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection