Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
arXiv:2010.02010
Abstract
We introduce Latent Meaning Cells, a deep latent variable model which learns contextualized representations of words by combining local lexical context and metadata. Metadata can refer to granular context, such as section type, or to more global context, such as unique document ids. Reliance on metadata for contextualized representation learning is apropos in the clinical domain where text is semi-structured and expresses high variation in topics. We evaluate the LMC model on the task of zero-shot clinical acronym expansion across three datasets. The LMC significantly outperforms a diverse set of baselines at a fraction of the pre-training cost and learns clinically coherent representations. We demonstrate that not only is metadata itself very helpful for the task, but that the LMC inference algorithm provides an additional large benefit.
arXiv commentTo appear in Proceedings Track for Machine Learning for Health (ML4H) Workshop at NeurIPS (2020)
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Sentences and Documents
- Publicly Available Clinical BERT Embeddings
- ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- Gradient Estimation Using Stochastic Computation Graphs
- Word Representations via Gaussian Embedding
- Breaking Sticks and Ambiguities with Adaptive Skip-gram
- Modeling Documents with Deep Boltzmann Machines
- Probabilistic FastText for Multi-Sense Word Embeddings
- Probing Biomedical Embeddings from Language Models
- A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
- Deep Contextualized Biomedical Abbreviation Expansion
- Training without training data: Improving the generalizability of automated medical abbreviation disambiguation