Distilled Wasserstein Learning for Word Embedding and Topic Modeling

Abstract

We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of topics, their optimal transports to the word distributions of documents, and the embeddings of words are learned in a unified framework. When learning the topic model, we leverage a distilled underlying distance matrix to update the topic distributions and smoothly calculate the corresponding optimal transports. Such a strategy provides the updating of word embeddings with robust guidance, improving the algorithmic convergence. As an application, we focus on patient admission records, in which the proposed method embeds the codes of diseases and procedures and learns the topics of admissions, obtaining superior performance on clinically-meaningful disease network construction, mortality prediction as a function of admission codes, and procedure recommendation.

Cited by in corpus (19)

• Gromov-Wasserstein Learning for Graph Matching and Node Embedding
• Discriminative Topic Mining via Category-Name Guided Text Embedding
• Topic-Guided Variational Autoencoders for Text Generation
• Hierarchical Optimal Transport for Document Representation
• Natural language processing of MIMIC-III clinical notes for identifying diagnosis and procedures with neural networks
• The Dynamic Embedded Topic Model
• Improving Sequence-to-Sequence Learning via Optimal Transport
• Re-evaluating Word Mover's Distance
• TopNet: Learning from Neural Topic Model to Generate Long Stories
• Statistical Optimal Transport posed as Learning Kernel Embedding
• Neural Topic Model via Optimal Transport
• Hierarchical Optimal Transport for Robust Multi-View Learning
• Optimal Transport for Stationary Markov Chains via Policy Iteration
• Interpretable ICD Code Embeddings with Self- and Mutual-Attention Mechanisms
• Text Mining Undergraduate Engineering Programs' Applications: the Role of Gender, Nationality, and Socio-economic Status
• Gaussian Hierarchical Latent Dirichlet Allocation: Bringing Polysemy Back
• Fixed Support Tree-Sliced Wasserstein Barycenter
• Concentrated Document Topic Model
• Exclusive Topic Modeling