SuperSim: a test set for word similarity and relatedness in Swedish
arXiv:2104.05228
Abstract
Language models are notoriously difficult to evaluate. We release SuperSim, a large-scale similarity and relatedness test set for Swedish built with expert human judgments. The test set is composed of 1,360 word-pairs independently judged for both relatedness and similarity by five annotators. We evaluate three different models (Word2Vec, fastText, and GloVe) trained on two separate Swedish datasets, namely the Swedish Gigaword corpus and a Swedish Wikipedia dump, to provide a baseline for future comparison. We release the fully annotated test set, code, baseline models, and data.
arXiv commentAccepted at NoDaLiDa 2021
References in corpus (4)
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change
- Separated by an Un-common Language: Towards Judgment Language Informed Vector Space Modeling
- Corpora Compared: The Case of the Swedish Gigaword & Wikipedia Corpora