Learning Conceptual-Contextual Embeddings for Medical Text

Abstract

External knowledge is often useful for natural language understanding tasks. We introduce a contextual text representation model called Conceptual-Contextual (CC) embeddings, which incorporates structured knowledge into text representations. Unlike entity embedding methods, our approach encodes a knowledge graph into a context model. CC embeddings can be easily reused for a wide range of tasks just like pre-trained language models. Our model effectively encodes the huge UMLS database by leveraging semantic generalizability. Experiments on electronic health records (EHRs) and medical text processing benchmarks showed our model gives a major boost to the performance of supervised medical NLP tasks.

References in corpus (9)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Enhanced LSTM for Natural Language Inference
• ERNIE: Enhanced Representation through Knowledge Integration
• SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions
• Neural Document Embeddings for Intensive Care Patient Mortality Prediction
• Beyond Word Embeddings: Learning Entity and Concept Representations from Large Scale Knowledge Bases
• Lessons from Natural Language Inference in the Clinical Domain