54 citations · 107 across the 13 of their papers we have counts for
15 papers
Should you marginalize over possible tokenizations?
Nadezhda Chirkova, Germán Kruszewski, Jos Rozen +1
Autoregressive language models (LMs) map token sequences to probabilities. The usual practice for computing the probability of any character string (e.g. English sentences) is to f…
Parameter-Efficient Finetuning of Transformers for Source Code
Shamil Ayupov, Nadezhda Chirkova
Pretrained Transformers achieve state-of-the-art performance in various code-processing tasks but may be too large to be deployed. As software development tools often incorporate m…
Probing Pretrained Models of Source Code
Sergey Troshin, Nadezhda Chirkova
Deep learning models are widely used for solving challenging code processing tasks, such as code generation or code summarization. Traditionally, a specific model architecture was…
Machine Learning Methods for Spectral Efficiency Prediction in Massive MIMO Systems
Evgeny Bobrov, Sergey Troshin, Nadezhda Chirkova +4
Channel decoding, channel detection, channel assessment, and resource management for wireless multiple-input multiple-output (MIMO) systems are all examples of problems where machi…
On the Memorization Properties of Contrastive Learning
Ildus Sadrtdinov, Nadezhda Chirkova, Ekaterina Lobacheva
Memorization studies of deep neural networks (DNNs) help to understand what patterns and how do DNNs learn, and motivate improvements to DNN training approaches. In this work, we i…
On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova +2
Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surpr…
On the Embeddings of Variables in Recurrent Neural Networks for Source Code
Nadezhda Chirkova
Source code processing heavily relies on the methods widely used in natural language processing (NLP), but involves specifics that need to be taken into account to achieve higher q…
A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code
Nadezhda Chirkova, Sergey Troshin
There is an emerging interest in the application of natural language processing models to source code processing tasks. One of the major problems in applying deep learning to softw…
Empirical Study of Transformers for Source Code
Nadezhda Chirkova, Sergey Troshin
Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. I…
On Power Laws in Deep Ensembles
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan +1
Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classifi…
Deep Ensembles on a Fixed Memory Budget: One Wide Network or Several Thinner Ones?
Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov
One of the generally accepted views of modern deep learning is that increasing the number of parameters usually leads to better quality. The two easiest ways to increase the number…
Structured Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Alexander Markovich +1
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification…
Bayesian Sparsification of Gated Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop…
Bayesian Compression for Natural Language Processing
Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these p…
Bayesian Sparsification of Recurrent Neural Networks
Ekaterina Lobacheva, Nadezhda Chirkova, Dmitry Vetrov
Recurrent neural networks show state-of-the-art results in many text analysis tasks but often require a lot of memory to store their weights. Recently proposed Sparse Variational D…