Evaluating Word Embedding Models: Methods and Experimental Results

Abstract

Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evalutors.

13 pages

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Neural Machine Translation by Jointly Learning to Align and Translate
• Natural Language Processing (almost) from Scratch
• Cross-lingual Language Model Pretraining
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• Unsupervised Statistical Machine Translation
• Introduction to the CoNLL-2000 Shared Task: Chunking
• A Survey of Word Embeddings Evaluation Methods
• Semantic Structure and Interpretability of Word Embeddings
• Automated Generation of Multilingual Clusters for the Evaluation of Distributed Representations

Cited by in corpus (19)

• Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes
• Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey
• Regional Differences in Information Privacy Concerns After the Facebook-Cambridge Analytica Data Scandal
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• Improved management of issue dependencies in issue trackers of large collaborative projects
• Summarising Historical Text in Modern Languages
• Op2Vec: An Opcode Embedding Technique and Dataset Design for End-to-End Detection of Android Malware
• Robust and Consistent Estimation of Word Embedding for Bangla Language by fine-tuning Word2Vec Model
• A Comprehensive Analysis of Static Word Embeddings for Turkish
• Word2Vec: Optimal Hyper-Parameters and Their Impact on NLP Downstream Tasks
• Predicting Vulnerability In Large Codebases With Deep Code Representation
• Inductive Learning on Commonsense Knowledge Graph Completion
• Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
• Hybrid deep learning methods for phenotype prediction from clinical notes
• Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training
• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality
• An Ensemble Method for Producing Word Representations focusing on the Greek Language
• Efficient Sentence Embedding via Semantic Subspace Analysis