Temporal Self-Attention Network for Medical Concept Embedding

Abstract

In longitudinal electronic health records (EHRs), the event records of a patient are distributed over a long period of time and the temporal relations between the events reflect sufficient domain knowledge to benefit prediction tasks such as the rate of inpatient mortality. Medical concept embedding as a feature extraction method that transforms a set of medical concepts with a specific time stamp into a vector, which will be fed into a supervised learning algorithm. The quality of the embedding significantly determines the learning performance over the medical data. In this paper, we propose a medical concept embedding method based on applying a self-attention mechanism to represent each medical concept. We propose a novel attention mechanism which captures the contextual information and temporal relationships between medical concepts. A light-weight neural net, "Temporal Self-Attention Network (TeSAN)", is then proposed to learn medical concept embedding based solely on the proposed attention mechanism. To test the effectiveness of our proposed methods, we have conducted clustering and prediction tasks on two public EHRs datasets comparing TeSAN against five state-of-the-art embedding methods. The experimental results demonstrate that the proposed TeSAN model is superior to all the compared methods. To the best of our knowledge, this work is the first to exploit temporal self-attentive relations between medical events.

10 pages, 7 figures, accepted at IEEE ICDM 2019

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Machine Translation by Jointly Learning to Align and Translate
• Scalable and accurate deep learning for electronic health records
• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• A Structured Self-attentive Sentence Embedding
• End-To-End Memory Networks
• Learning Natural Language Inference using Bidirectional LSTM model and Inner-Attention
• MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Reinforced Mnemonic Reader for Machine Reading Comprehension

Cited by in corpus (6)

• Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies
• CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks
• Multi-Time Attention Networks for Irregularly Sampled Time Series
• Self-Attention Enhanced Patient Journey Understanding in Healthcare System
• Sequential Diagnosis Prediction with Transformer and Ontological Representation
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes