Analysing Lexical Semantic Change with Contextualised Word Representations

Abstract

This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural language model to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.

arXiv commentTo appear in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL-2020)

References in corpus (8)

• Semi-supervised Sequence Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Dynamic Word Embeddings
• Diachronic word embeddings and semantic shifts: a survey
• Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings
• "I don't believe in word senses"
• WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
• Capturing Evolution in Word Usage: Just Add More Clusters?

Cited by in corpus (16)

• Time-Aware Language Models as Temporal Knowledge Bases
• Capturing Evolution in Word Usage: Just Add More Clusters?
• A Survey on Contextualised Semantic Shift Detection
• Statistically significant detection of semantic shifts using contextual word embeddings
• Challenges for Computational Lexical Semantic Change
• Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses
• Characterizing English Variation across Social Media Communities with BERT
• Enriching Word Embeddings with Temporal and Spatial Information
• Survey in Characterizing Semantic Change
• Time Masking for Temporal Language Models
• A diachronic evaluation of gender asymmetry in euphemism
• Predicting emergent linguistic compositions through time: Syntactic frame extension via multimodal chaining
• Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings
• On the Universality of Deep Contextual Language Models
• Text analysis and deep learning: A network approach
• Grammatical Profiling for Semantic Change Detection