2.3k citations · 2.9k across the 31 of their papers we have counts for
34 papers
Graph Sensitivity of Cartesian Products with Matched Bridges
Zhen-Mu Hong, Zi-Yi Wu, Zheng-Jiang Xia
For a graph , let denote the minimum of the maximum degree of an induced subgraph with vertices, where is the independence number, and write $f(G)=f_1(G…
Sharpening Tax in Post-Training
Changdae Oh, Qi Zeng, Qi Qi +7
An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving sing…
LoRA-generating hypernetworks for efficient on-device LLM generative personalization
Sean Augenstein, Li Ding, Jihwan Lee +2
On-device large language models (`LLMs'), e.g. running on mobile phones, are ripe for improvement via personalization. The limited compute resources of mobile devices impose limits…
Contextually Guided Transformers via Low-Rank Adaptation
Andrey Zhmoginov, Jihwan Lee, Max Vladymyrov +1
Large Language Models (LLMs) based on Transformers excel at text processing, but their reliance on prompts for specialized behavior introduces computational overhead. We propose a…
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
Andrey Zhmoginov, Jihwan Lee, Mark Sandler
Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very…
How new data permeates LLM knowledge and how to dilute it
Chen Sun, Renat Aksitov, Andrey Zhmoginov +5
Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leadin…
Long Context In-Context Compression by Getting to the Gist of Gisting
Aleksandar Petrov, Mark Sandler, Andrey Zhmoginov +2
Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-co…
Learning and Unlearning of Fabricated Knowledge in Language Models
Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov +2
What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this…
MELODI: Exploring Memory Compression for Long Contexts
Yinpeng Chen, DeLesley Hutchins, Aren Jansen +3
We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-ter…
Narrowing the Focus: Learned Optimizers for Pretrained Models
Gus Kristiansen, Mark Sandler, Andrey Zhmoginov +4
In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand…
Training trajectories, mini-batch losses and the curious role of the learning rate
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov +1
Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. I…
Continual HyperTransformer: A Meta-Learner for Continual Few-Shot Learning
Max Vladymyrov, Andrey Zhmoginov, Mark Sandler
We focus on the problem of learning without forgetting from multiple tasks arriving sequentially, where each task is defined using a few-shot episode of novel or already seen class…
Transformers learn in-context by gradient descent
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo +4
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on au…
Decentralized Learning with Multi-Headed Distillation
Andrey Zhmoginov, Mark Sandler, Nolan Miller +2
Decentralized learning with private data is a central problem in machine learning. We propose a novel distillation-based decentralized learning technique that allows multiple agent…
Fine-tuning Image Transformers using Learnable Memory
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov +1
In this paper we propose augmenting Vision Transformer models with learnable memory tokens. Our approach allows the model to adapt to new tasks, using few parameters, while optiona…
HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learning
Andrey Zhmoginov, Mark Sandler, Max Vladymyrov
In this work we propose a HyperTransformer, a Transformer-based model for supervised and semi-supervised few-shot learning that generates weights of a convolutional neural network…
Compositional Models: Multi-Task Learning and Knowledge Transfer with Modular Networks
Andrey Zhmoginov, Dina Bashkirova, Mark Sandler
Conditional computation and modular networks have been recently proposed for multitask learning and other problems as a way to decompose problem solving into multiple reusable comp…
Limit on the Electric Charge of Antihydrogen
A. Capra, C. Amole, M. D. Ashkezari +37
The ALPHA collaboration has successfully demonstrated the production and the confinement of cold antihydrogen, . An analysis of trapping data allowed a strin…
BasisNet: Two-stage Model Synthesis for Efficient Inference
Mingda Zhang, Chun-Te Chu, Andrey Zhmoginov +6
In this work, we present BasisNet which combines recent advancements in efficient neural network architectures, conditional computation, and early termination in a simple new form.…
Meta-Learning Bidirectional Update Rules
Mark Sandler, Max Vladymyrov, Andrey Zhmoginov +4
In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in n…
Large-Scale Generative Data-Free Distillation
Liangchen Luo, Mark Sandler, Zi Lin +2
Knowledge distillation is one of the most popular and effective techniques for knowledge transfer, model compression and semi-supervised learning. Most existing distillation approa…
Image segmentation via Cellular Automata
Mark Sandler, Andrey Zhmoginov, Liangcheng Luo +3
In this paper, we propose a new approach for building cellular automata to solve real-world segmentation problems. We design and train a cellular automaton that can successfully se…
Non-discriminative data or weak model? On the relative importance of data and model resolution
Mark Sandler, Jonathan Baccash, Andrey Zhmoginov +1
We explore the question of how the resolution of the input image ("input resolution") affects the performance of a neural network when compared to the resolution of the hidden laye…
Information-Bottleneck Approach to Salient Region Discovery
Andrey Zhmoginov, Ian Fischer, Mark Sandler
We propose a new method for learning image attention masks in a semi-supervised setting based on the Information Bottleneck principle. Provided with a set of labeled images, the ma…