On the Dimensionality of Word Embedding

Abstract

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

arXiv comment18 pages, Advances in Neural Information Processing Systems 31 (NeurIPS 2018, Oral Presentation)

References in corpus (9)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Empirical Analysis of Predictive Algorithms for Collaborative Filtering
• From Frequency to Meaning: Vector Space Models of Semantics
• Exploiting Similarities among Languages for Machine Translation
• Matrix estimation by Universal Singular Value Thresholding
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

Cited by in corpus (36)

• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Cultural Cartography with Word Embeddings
• Principled approach to the selection of the embedding dimension of networks
• DDGK: Learning Graph Representations for Deep Divergence Graph Kernels
• Quantum Language Model with Entanglement Embedding for Question Answering
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• The Shape of Data: Intrinsic Distance for Data Distributions
• Temporal Aggregation and Propagation Graph Neural Networks for Dynamic Representation
• Multi-criteria recommendation systems to foster online grocery
• Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems
• On the Dimensionality of Embeddings for Sparse Features and Data
• TiM4Rec: An Efficient Sequential Recommendation Model Based on Time-Aware Structured State Space Duality Model
• Sequence embeddings help to identify fraudulent cases in healthcare insurance
• Filtered Inner Product Projection for Crosslingual Embedding Alignment
• Evaluation of taxonomic and neural embedding methods for calculating semantic similarity
• Understanding the Downstream Instability of Word Embeddings
• Effects of Number of Filters of Convolutional Layers on Speech Recognition Model Accuracy
• Training with Multi-Layer Embeddings for Model Reduction
• Word2vec Skip-gram Dimensionality Selection via Sequential Normalized Maximum Likelihood
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction
• Parallel Computation of Graph Embeddings
• Sparse Lifting of Dense Vectors: Unifying Word and Sentence Representations
• Multilingual Embeddings Jointly Induced from Contexts and Concepts: Simple, Strong and Scalable
• Decompressing Knowledge Graph Representations for Link Prediction
• Deep Joint Embeddings of Context and Content for Recommendation
• On the Downstream Performance of Compressed Word Embeddings
• Real-World Networks are Low-Dimensional: Theoretical and Practical Assessment
• On Dimensional Linguistic Properties of the Word Embedding Space
• Blind signal decomposition of various word embeddings based on join and individual variance explained
• SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for Domain-specific Small Corpus
• An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek
• Learning Word Embeddings with Domain Awareness
• Single Training Dimension Selection for Word Embedding with PCA
• Word Embeddings: Stability and Semantic Change
• RPD: A Distance Function Between Word Embeddings
• Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Models