Sparse Overcomplete Word Vector Representations

Abstract

Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Proceedings of ACL 2015

References in corpus (1)

• From Frequency to Meaning: Vector Space Models of Semantics

Cited by in corpus (7)

• Understanding Neural Networks through Representation Erasure
• Visualizing and Understanding Neural Models in NLP
• xSense: Learning Sense-Separated Sparse Representations and Textual Definitions for Explainable Word Sense Networks
• Arabic Named Entity Recognition using Word Representations
• Sparse Lifting of Dense Vectors: Unifying Word and Sentence Representations
• Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings
• Learning and Evaluating Sparse Interpretable Sentence Embeddings