604 citations · 2.5k across the 63 of their papers we have counts for
27 papers · 1 filter
Beyond Modality Fusion: Deep Ensembles for Multimodal Classification
Ilya Burenko, Dmitry Vetrov
In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced,…
To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
Ildus Sadrtdinov, Dmitrii Pozdeev, Dmitry Vetrov +1
Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of mode…
Entropic Neural Optimal Transport via Diffusion Processes
Nikita Gushchin, Alexander Kolesov, Alexander Korotin +2
We propose a novel neural algorithm for the fundamental problem of computing the entropic optimal transport (EOT) plan between continuous probability distributions which are access…
Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
Maxim Kodryan, Ekaterina Lobacheva, Maksim Nakhodnov +1
A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant. The intrinsic domain of s…
Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces
Kirill Struminsky, Artyom Gadetsky, Denis Rakitin +2
Structured latent variables allow incorporating meaningful prior knowledge into deep learning models. However, learning with such variables remains challenging because of their dis…
Automating Control of Overestimation Bias for Reinforcement Learning
Arsenii Kuznetsov, Alexander Grishin, Artem Tsypin +3
Overestimation bias control techniques are used by the majority of high-performing off-policy reinforcement learning algorithms. However, most of these techniques rely on pre-defin…
Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations
Arsenii Ashukha, Andrei Atanov, Dmitry Vetrov
Averaging predictions over a set of models -- an ensemble -- is widely used to improve predictive performance and uncertainty estimation of deep learning models. At the same time,…
On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova +2
Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surpr…
Towards Practical Credit Assignment for Deep Reinforcement Learning
Vyacheslav Alipov, Riley Simmons-Edler, Nikita Putintsev +2
Credit assignment is a fundamental problem in reinforcement learning, the problem of measuring an action's influence on future rewards. Explicit credit assignment methods have the…
On Power Laws in Deep Ensembles
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan +1
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classifi…
Involutive MCMC: a Unifying Framework
Kirill Neklyudov, Max Welling, Evgenii Egorov +1
Markov Chain Monte Carlo (MCMC) is a computational approach to fundamental problems such as inference, integration, optimization, and simulation. The field has developed a broad sp…
MARS: Masked Automatic Ranks Selection in Tensor Decompositions
Maxim Kodryan, Dmitry Kropotov, Dmitry Vetrov
Tensor decomposition methods have proven effective in various applications, including compression and acceleration of neural networks. At the same time, the problem of determining…
Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?
Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov
One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number…
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin +1
The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous co…
Deterministic Decoding for Discrete Data in Variational Autoencoders
Daniil Polykovskiy, Dmitry Vetrov
Variational autoencoders are prominent generative models for modeling discrete data. However, with flexible decoders, they tend to ignore the latent codes. In this paper, we study…
Stochasticity in Neural ODEs: An Empirical Study
Viktor Oganesyan, Alexandra Volokhova, Dmitry Vetrov
Stochastic regularization of neural networks (e.g. dropout) is a wide-spread technique in deep learning that allows for better generalization. Despite its success, continuous-time…
Low-variance Black-box Gradient Estimates for the Plackett-Luce Distribution
Artyom Gadetsky, Kirill Struminsky, Christopher Robinson +2
Learning models with discrete latent variables using stochastic gradient descent remains a challenge due to the high variance of gradient estimates. Modern variance reduction techn…
Structured Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Alexander Markovich +1
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification…
A Prior of a Googol Gaussians: a Tensor Ring Induced Prior for Generative Models
Maksim Kuznetsov, Daniil Polykovskiy, Dmitry Vetrov +1
Generative models produce realistic objects in many domains, including text, image, video, and audio synthesis. Most popular models---Generative Adversarial Networks (GANs) and Var…
Subspace Inference for Bayesian Deep Learning
Pavel Izmailov, Wesley J. Maddox, Polina Kirichenko +3
Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Ba…
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Wesley Maddox, Timur Garipov, Pavel Izmailov +2
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA),…
Bayesian Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop…
Averaging Weights Leads to Wider Optima and Better Generalization
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov +2
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple a…
Probabilistic Adaptive Computation Time
Michael Figurnov, Artem Sobolev, Dmitry Vetrov
We present a probabilistic model with discrete latent variables that control the computation time in deep learning models such as ResNets and LSTMs. A prior on the latent variables…