4 citations · 28 across the 86 of their papers we have counts for
27 papers · 1 filter
Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference
Egor Sevriugov, Nikita Dragunov, Anton Razzhigaev +2
Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately mode…
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev +3
The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model beco…
Tensor-Train Point Cloud Compression and Efficient Approximate Nearest-Neighbor Search
Georgii Novikov, Alexander Gneushev, Alexey Kadeishvili +1
Nearest-neighbor search in large vector databases is crucial for various machine learning applications. This paper introduces a novel method using tensor-train (TT) low-rank tensor…
CLEAR: Character Unlearning in Textual and Visual Modalities
Alexey Dontsov, Dmitrii Korzh, Alexey Zhavoronkin +6
Machine Unlearning (MU) is critical for removing private or hazardous information from deep learning models. While MU has advanced significantly in unimodal (text or vision) settin…
Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition
Artem Basharin, Andrei Chertkov, Ivan Oseledets
We propose a new model for multi-token prediction in transformers, aiming to enhance sampling efficiency without compromising accuracy. Motivated by recent work that predicts the p…
Associative memory and dead neurons
Vladimir Fanaskov, Ivan Oseledets
In "Large Associative Memory Problem in Neurobiology and Machine Learning," Dmitry Krotov and John Hopfield introduced a general technique for the systematic construction of neural…
Anatomical Positional Embeddings
Mikhail Goncharov, Valentin Samokhin, Eugenia Soboleva +5
We propose a self-supervised model producing 3D anatomical positional embeddings (APE) of individual medical image voxels. APE encodes voxels' anatomical closeness, i.e., voxels of…
Spectral Informed Neural Network: An Efficient and Low-Memory PINN
Tianchi Yu, Yiming Qi, Ivan Oseledets +1
With growing investigations into solving partial differential equations by physics-informed neural networks (PINNs), more accurate and efficient PINNs are required to meet the prac…
Inverted Activations: Reducing Memory Footprint in Neural Network Training
Georgii Novikov, Ivan Oseledets
The scaling of neural networks with increasing data and model sizes necessitates the development of more efficient deep learning algorithms. A significant challenge in neural netwo…
Data-driven optimal prediction with control
Aleksandr Katrutsa, Ivan Oseledets, Sergey Utyuzhnikov
This study presents the extension of the data-driven optimal prediction approach to the dynamical system with control. The optimal prediction is used to analyze dynamical systems i…
ConDiff: A Challenging Dataset for Neural Solvers of Partial Differential Equations
Vladislav Trifonov, Alexander Rudikov, Oleg Iliev +3
We present ConDiff, a novel dataset for scientific machine learning. ConDiff focuses on the parametric diffusion equation with space dependent coefficients, a fundamental problem i…
Astral: training physics-informed neural networks with error majorants
Vladimir Fanaskov, Tianchi Yu, Alexander Rudikov +1
The primal approach to physics-informed learning is a residual minimization. We argue that residual is, at best, an indirect measure of the error of approximate solution and propos…
Your Transformer is Secretly Linear
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova +4
This paper reveals a novel linear characteristic exclusive to transformer decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We analyze embedding transformations…
GLiRA: Black-Box Membership Inference Attack via Knowledge Distillation
Andrey V. Galichin, Mikhail Pautov, Alexey Zhavoronkin +2
While Deep Neural Networks (DNNs) have demonstrated remarkable performance in tasks related to perception and control, there are still several unresolved concerns regarding the pri…
Learning from Linear Algebra: A Graph Neural Network Approach to Preconditioner Design for Conjugate Gradient Solvers
Vladislav Trifonov, Alexander Rudikov, Oleg Iliev +3
Large linear systems are ubiquitous in modern computational science and engineering. The main recipe for solving them is the use of Krylov subspace iterative methods with well-desi…
Quantization of Large Language Models with an Overdetermined Basis
Daniil Merkulov, Daria Cherniuk, Alexander Rudikov +4
In this paper, we introduce an algorithm for data quantization based on the principles of Kashin representation. This approach hinges on decomposing any given vector, matrix, or te…
OmniFusion Technical Report
Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk +6
Last year, multimodal architectures served up a revolution in AI-based approaches and solutions, extending the capabilities of large language models (LLM). We propose an \textit{Om…
Certification of Speaker Recognition Models to Additive Perturbations
Dmitrii Korzh, Elvir Karimov, Mikhail Pautov +2
Speaker recognition technology is applied to various tasks, from personal virtual assistants to secure access systems. However, the robustness of these systems against adversarial…
Neural operators meet conjugate gradients: The FCG-NO method for efficient PDE solving
Alexander Rudikov, Vladimir Fanaskov, Ekaterina Muravleva +2
Deep learning solvers for partial differential equations typically have limited accuracy. We propose to overcome this problem by using them as preconditioners. More specifically, w…
Neural functional a posteriori error estimates
Vladimir Fanaskov, Alexander Rudikov, Ivan Oseledets
We propose a new loss function for supervised and physics-informed training of neural networks and operators that incorporates a posteriori error estimate. More specifically, durin…
Black-Box Approximation and Optimization with Hierarchical Tucker Decomposition
Gleb Ryzhakov, Andrei Chertkov, Artem Basharin +1
We develop a new method HTBB for the multidimensional black-box approximation and gradient-free optimization, which is based on the low-rank hierarchical Tucker decomposition with…
LoTR: Low Tensor Rank Weight Adaptation
Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev +2
In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fin…
Explicit Flow Matching: On The Theory of Flow Matching Algorithms with Applications
Gleb Ryzhakov, Svetlana Pavlova, Egor Sevriugov +1
This paper proposes a novel method, Explicit Flow Matching (ExFM), for training and analyzing flow-based generative models. ExFM leverages a theoretically grounded loss function, E…
TQCompressor: improving tensor decomposition methods in neural networks via permutations
V. Abronin, A. Naumov, D. Mazur +7
We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage…