Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling

Abstract

A common evaluation practice in the vector space models (VSMs) literature is to measure the models' ability to predict human judgments about lexical semantic relations between word pairs. Most existing evaluation sets, however, consist of scores collected for English word pairs only, ignoring the potential impact of the judgment language in which word pairs are presented on the human scores. In this paper we translate two prominent evaluation sets, wordsim353 (association) and SimLex999 (similarity), from English to Italian, German and Russian and collect scores for each dataset from crowdworkers fluent in its language. Our analysis reveals that human judgments are strongly impacted by the judgment language. Moreover, we show that the predictions of monolingual VSMs do not necessarily best correlate with human judgments made with the language used for model training, suggesting that models and humans are affected differently by the language they use when making semantic judgments. Finally, we show that in a large number of setups, multilingual VSM combination results in improved correlations with human judgments, suggesting that multilingualism may partially compensate for the judgment language effect on human judgments.

References in corpus (6)

• Natural Language Processing (almost) from Scratch
• From Frequency to Meaning: Vector Space Models of Semantics
• An Autoencoder Approach to Learning Bilingual Word Representations
• Multilingual Models for Compositional Distributed Semantics
• Learning Bilingual Word Representations by Marginalizing Alignments
• Learning Multilingual Word Representations using a Bag-of-Words Autoencoder

Cited by in corpus (12)

• Massively Multilingual Word Embeddings
• A Survey Of Cross-lingual Word Embedding Models
• Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints
• Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals
• Improving Cross-Lingual Word Embeddings by Meeting in the Middle
• Survey on the Use of Typological Information in Natural Language Processing
• SuperSim: a test set for word similarity and relatedness in Swedish
• Language classification from bilingual word embedding graphs
• Enhancing Word Embeddings with Knowledge Extracted from Lexical Resources
• Analysis Methods in Neural Language Processing: A Survey
• Proximal Mapping for Deep Regularization
• PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding