54 citations · 107 across the 13 of their papers we have counts for
7 papers · 1 filter
On the Memorization Properties of Contrastive Learning
Ildus Sadrtdinov, Nadezhda Chirkova, Ekaterina Lobacheva
Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we i…
On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova +2
Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surpr…
Empirical Study of Transformers for Source Code
Nadezhda Chirkova, Sergey Troshin
Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. I…
On Power Laws in Deep Ensembles
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan +1
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classifi…
Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?
Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov
One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number…
Structured Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Alexander Markovich +1
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification…
Bayesian Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop…