20 citations · 31 across the 18 of their papers we have counts for
22 papers
Posterior Frequencies and the Cost of Not Fitting: A Correction to a Memorization Lower Bound
Zhenisbek Assylbekov, Rustem Takhanov
Feldman's memorization lower bound assigns a frequency coefficient based on how often a point appears in the training sample. We give an exact three-point counterexample to its con…
Structure of kissing arrangements in and a place for the st sphere
Rustem Takhanov, Zhenisbek Assylbekov, Stanislav Yun
Most currently known kissing arrangements of size in share a common structure. They consist of vectors supported on , an…
Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding
Rustem Takhanov, Zhenisbek Assylbekov
Conditionally positive definite (CPD) kernels are defined with respect to a function class . It is well known that such a kernel is associated with its native spac…
Simplex Deep Linear Discriminant Analysis
Maxat Tezekbayev, Arman Bolatov, Zhenisbek Assylbekov
We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attachi…
Deep Linear Discriminant Analysis Revisited
Maxat Tezekbayev, Rustem Takhanov, Arman Bolatov +1
We show that for unconstrained Deep Linear Discriminant Analysis (LDA) classifiers, maximum-likelihood training admits pathological solutions in which class means drift together, c…
Overspecified Mixture Discriminant Analysis: Exponential Convergence, Statistical Guarantees, and Remote Sensing Applications
Arman Bolatov, Alan Legg, Igor Melnykov +3
This study explores the classification error of Mixture Discriminant Analysis (MDA) in scenarios where the number of mixture components exceeds those present in the actual data dis…
Learning Overspecified Gaussian Mixtures Exponentially Fast with the EM Algorithm
Zhenisbek Assylbekov, Alan Legg, Artur Pak
We investigate the convergence properties of the EM algorithm when applied to overspecified Gaussian mixture models -- that is, when the number of components in the fitted model ex…
Intractability of Learning the Discrete Logarithm with Gradient-Based Methods
Rustem Takhanov, Maxat Tezekbayev, Artur Pak +3
The discrete logarithm problem is a fundamental challenge in number theory with significant implications for cryptographic protocols. In this paper, we investigate the limitations…
Gradient Descent Fails to Learn High-frequency Functions and Modular Arithmetic
Rustem Takhanov, Maxat Tezekbayev, Artur Pak +2
Classes of target functions containing a large number of approximately orthogonal elements are known to be hard to learn by the Statistical Query algorithms. Recently this classica…
Long-Tail Theory under Gaussian Mixtures
Arman Bolatov, Maxat Tezekbayev, Igor Melnykov +3
We suggest a simple Gaussian mixture model for data generation that complies with Feldman's long tail theory (2020). We demonstrate that a linear classifier cannot decrease the gen…
From Hyperbolic Geometry Back to Word Embeddings
Sultan Nurmukhamedov, Thomas Mach, Arsen Sheverdin +1
We choose random points in the hyperbolic disc and claim that these points are already word representations. However, it is yet to be uncovered which point corresponds to which wor…
Speeding Up Entmax
Maxat Tezekbayev, Vassilina Nikoulina, Matthias Gallé +1
Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits. However, by producing a dense probability distribution each…
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
Vassilina Nikoulina, Maxat Tezekbayev, Nuradil Kozhakhmet +3
There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether lin…
Squashed Shifted PMI Matrix: Bridging Word Embeddings and Hyperbolic Spaces
Zhenisbek Assylbekov, Alibi Jangeldin
We show that removing sigmoid transformation in the skip-gram with negative sampling (SGNS) objective does not harm the quality of word vectors significantly and at the same time i…
Semantics- and Syntax-related Subvectors in the Skip-gram Embeddings
Maxat Tezekbayev, Zhenisbek Assylbekov, Rustem Takhanov
We show that the skip-gram embedding of any word can be decomposed into two subvectors which roughly correspond to semantic and syntactic roles of the word.
A Critique of the Smooth Inverse Frequency Sentence Embeddings
Aidana Karipbayeva, Alena Sorokina, Zhenisbek Assylbekov
We critically review the smooth inverse frequency sentence embedding method of Arora, Liang, and Ma (2017), and show inconsistencies in its setup, derivation, and evaluation.
Low-Rank Approximation of Matrices for PMI-based Word Embeddings
Alena Sorokina, Aidana Karipbayeva, Zhenisbek Assylbekov
We perform an empirical evaluation of several methods of low-rank approximation in the problem of obtaining PMI-based word embeddings. All word vectors were trained on parts of a l…
Context Vectors are Reflections of Word Vectors in Half the Dimensions
Zhenisbek Assylbekov, Rustem Takhanov
This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilisti…
Fourier Neural Networks: A Comparative Study
Abylay Zhumekenov, Malika Uteuliyeva, Olzhas Kabdolov +3
We review neural network architectures which were motivated by Fourier series and integrals and which are referred to as Fourier neural networks. These networks are empirically eva…
Reusing Weights in Subword-aware Neural Language Models
Zhenisbek Assylbekov, Rustem Takhanov
We propose several ways of reusing subword embeddings and other weights in subword-aware neural language models. The proposed techniques do not benefit a competitive character-awar…
Patterns versus Characters in Subword-aware Neural Language Modeling
Rustem Takhanov, Zhenisbek Assylbekov
Words in some natural languages can have a composite structure. Elements of this structure include the root (that could also be composite), prefixes and suffixes with which various…
Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones
Zhenisbek Assylbekov, Rustem Takhanov, Bagdat Myrzakhmetov +1
Syllabification does not seem to improve word-level RNN language modeling quality when compared to character-based segmentation. However, our best syllable-aware language model, ac…