On the Dimensionality of Embeddings for Sparse Features and Data

Abstract

In this note we discuss a common misconception, namely that embeddings are always used to reduce the dimensionality of the item space. We show that when we measure dimensionality in terms of information entropy then the embedding of sparse probability distributions, that can be used to represent sparse features or data, may or not reduce the dimensionality of the item space. However, the embeddings do provide a different and often more meaningful representation of the items for a particular task at hand. Also, we give upper bounds and more precise guidelines for choosing the embedding dimension.

8 pages, 2 figures

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Opening the Black Box of Deep Neural Networks via Information
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• On the Dimensionality of Word Embedding

Cited by in corpus (6)

• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems
• Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation
• Time-based Sequence Model for Personalization and Recommendation Systems
• Training with Multi-Layer Embeddings for Model Reduction