A Survey On Neural Word Embeddings

Abstract

Understanding human language has been a sub-challenge on the way of intelligent machines. The study of meaning in natural language processing (NLP) relies on the distributional hypothesis where language elements get meaning from the words that co-occur within contexts. The revolutionary idea of distributed representation for a concept is close to the working of a human mind in that the meaning of a word is spread across several neurons, and a loss of activation will only slightly affect the memory retrieval process. Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks. In this survey, we provide a comprehensive literature review on neural word embeddings. We give theoretical foundations and describe existing work by an interplay between word embeddings and language modelling. We provide broad coverage on neural word embeddings, including early word embeddings, embeddings targeting specific semantic relations, sense embeddings, morpheme embeddings, and finally, contextual representations. Finally, we describe benchmark datasets in word embeddings' performance evaluation and downstream tasks along with the performance results of/due to word embeddings.

arXiv comment33 pages, 2 figures, 8 tables

References in corpus (15)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Natural Language Processing (almost) from Scratch
• Language Models are Few-Shot Learners
• From Frequency to Meaning: Vector Space Models of Semantics
• Character-Aware Neural Language Models
• ERNIE: Enhanced Representation through Knowledge Integration
• An Empirical Study of Smoothing Techniques for Language Modeling
• Semi-supervised Sequence Learning
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• Learned in Translation: Contextualized Word Vectors
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Compositional Morphology for Word Representations and Language Modelling
• Diachronic word embeddings and semantic shifts: a survey
• A Survey of Word Embeddings Evaluation Methods