LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
arXiv:2105.12449 · doi:10.1016/j.artint.2022.103661
Abstract
Distributional semantics based on neural approaches is a cornerstone of Natural Language Processing, with surprising connections to human meaning representation as well. Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information, simply as a product of self-supervision. Prior work has shown that these contextual representations can be used to accurately represent large sense inventories as sense embeddings, to the extent that a distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms models trained specifically for the task. Still, there remains much to understand on how to use these Neural Language Models (NLMs) to produce sense embeddings that can better harness each NLM's meaning representation abilities. In this work we introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We also emphasize the versatility of these sense embeddings in contrast to task-specific models, applying them on several sense-related tasks, besides WSD, while demonstrating improved performance using our proposed approach over prior work focused on sense embeddings. Finally, we discuss unexpected findings regarding layer and model performance variations, and potential applications for downstream tasks.
arXiv commentAccepted to Artificial Intelligence Journal (AIJ)
References in corpus (14)
- Distributed Representations of Words and Phrases and their Compositionality
- Natural Language Processing (almost) from Scratch
- Language Models are Few-Shot Learners
- On Calibration of Modern Neural Networks
- ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
- Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
- Deep Recurrent Quantization for Generating Sequential Binary Codes
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes
- Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
- What do you learn from context? Probing for sentence structure in contextualized word representations
- Semi-supervised Word Sense Disambiguation with Neural Models
- CoSimLex: A Resource for Evaluating Graded Word Similarity in Context
- Challenges in Automated Debiasing for Toxic Language Detection