20 citations · 31 across the 18 of their papers we have counts for
10 papers · 1 filter
From Hyperbolic Geometry Back to Word Embeddings
Sultan Nurmukhamedov, Thomas Mach, Arsen Sheverdin +1
We choose random points in the hyperbolic disc and claim that these points are already word representations. However, it is yet to be uncovered which point corresponds to which wor…
Speeding Up Entmax
Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé +1
Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits. However, by producing a dense probability distribution each…
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet +3
There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether lin…
Squashed Shifted PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces
Zhenisbek Assylbekov, Alibi Jangeldin
We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time i…
Semantics- and Syntax-related Subvectors in the Skip-gram Embeddings
Maxat Tezekbayev, Zhenisbek Assylbekov, Rustem Takhanov
We show that the skip-gram embedding of any word can be decomposed into two subvectors which roughly correspond to semantic and syntactic roles of the word.
A Critique of the Smooth Inverse Frequency Sentence Embeddings
Aidana Karipbayeva, Alena Sorokina, Zhenisbek Assylbekov
We critically review the smooth inverse frequency sentence embedding method of Arora, Liang, and Ma (2017), and show inconsistencies in its setup, derivation, and evaluation.
Low-Rank Approximation of Matrices for PMI-based Word Embeddings
Alena Sorokina, Aidana Karipbayeva, Zhenisbek Assylbekov
We perform an empirical evaluation of several methods of low-rank approximation in the problem of obtaining PMI-based word embeddings. All word vectors were trained on parts of a l…
Reusing Weights in Subword-aware Neural Language Models
Zhenisbek Assylbekov, Rustem Takhanov
We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-awar…
Patterns versus Characters in Subword-aware Neural Language Modeling
Rustem Takhanov, Zhenisbek Assylbekov
Words in some natural languages can have a composite structure. Elements of this structure include the root (that could also be composite), prefixes and suffixes with which various…
Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones
Zhenisbek Assylbekov, Rustem Takhanov, Bagdat Myrzakhmetov +1
Syllabification does not seem to improve word-level RNN language modeling quality when compared to character-based segmentation. However, our best syllable-aware language model, ac…