Dynamic Word Embeddings
arXiv:1702.08359
Abstract
We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.
In the proceedings of the International Conference on Machine Learning (ICML 2017); 8 pages + references and supplement
References in corpus (2)
Cited by in corpus (37)
- Deep Learning for Anomaly Detection: A Survey
- Diachronic word embeddings and semantic shifts: a survey
- Analysing Lexical Semantic Change with Contextualised Word Representations
- Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change
- Survey of Computational Approaches to Lexical Semantic Change
- Representation Learning for Dynamic Graphs: A Survey
- Representation Learning over Dynamic Graphs
- Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
- GP-VAE: Deep Probabilistic Time Series Imputation
- Scalable Generalized Dynamic Topic Models
- The Dynamic Embedded Topic Model
- Advances in Variational Inference
- SFE-GACN: A Novel Unknown Attack Detection Method Using Intra Categories Generation in Embedding Space
- Dynamic Bernoulli Embeddings for Language Evolution
- "What's ur type?" Contextualized Classification of User Types in Marijuana-related Communications using Compositional Multiview Embedding
- Structured Black Box Variational Inference for Latent Time Series Models
- Temporal Embeddings and Transformer Models for Narrative Text Understanding
- Perturbative Black Box Variational Inference
- Tracing cultural diachronic semantic shifts in Russian using word embeddings: test sets and baselines
- Structured Embedding Models for Grouped Data
- dynnode2vec: Scalable Dynamic Network Embedding
- Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
- GRADE: Graph Dynamic Embedding
- Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning
- Improving Optimization for Models With Continuous Symmetry Breaking
- Time Masking for Temporal Language Models
- Enriching Word Embeddings with Temporal and Spatial Information
- Empirical Study of Diachronic Word Embeddings for Scarce Data
- Variational Bayesian Quantization
- Measuring diachronic sense change: new models and Monte Carlo methods for Bayesian inference
- A diachronic evaluation of gender asymmetry in euphemism
- UoB at SemEval-2020 Task 1: Automatic Identification of Novel Word Senses
- A Statistical Model of Word Rank Evolution
- Temporal Graph Offset Reconstruction: Towards Temporally Robust Graph Representation Learning
- An Improved Historical Embedding without Alignment
- Generating Timelines by Modeling Semantic Change
- On the Universality of Deep Contextual Language Models