92 citations · 98 across the 12 of their papers we have counts for
12 papers
Evaluation of Neural Surrogates for Physical Modelling Synthesis of Nonlinear Elastic Plates
Carlos De La Vega Martin, Rodrigo Diaz Fernandez, Mark Sandler
Physical modelling synthesis aims to generate audio from physical simulations of vibrating structures. Thin elastic plates are a common model for drum membranes. Traditional numeri…
Contextually Guided Transformers via Low-Rank Adaptation
Andrey Zhmoginov, Jihwan Lee, Max Vladymyrov +1
Large Language Models (LLMs) based on Transformers excel at text processing, but their reliance on prompts for specialized behavior introduces computational overhead. We propose a…
Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
Andrey Zhmoginov, Jihwan Lee, Mark Sandler
Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very…
Fast Differentiable Modal Simulation of Non-linear Strings, Membranes, and Plates
Rodrigo Diaz, Mark Sandler
Modal methods for simulating vibrations of strings, membranes, and plates are widely used in acoustics and physically informed audio synthesis. However, traditional implementations…
How new data permeates LLM knowledge and how to dilute it
Chen Sun, Renat Aksitov, Andrey Zhmoginov +5
Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leadin…
Long Context In-Context Compression by Getting to the Gist of Gisting
Aleksandar Petrov, Mark Sandler, Andrey Zhmoginov +2
Long context processing is critical for the adoption of LLMs, but existing methods often introduce architectural complexity that hinders their practical adoption. Gisting, an in-co…
Learning and Unlearning of Fabricated Knowledge in Language Models
Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov +2
What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this…
Towards Efficient Modelling of String Dynamics: A Comparison of State Space and Koopman based Deep Learning Methods
Rodrigo Diaz, Carlos De La Vega Martin, Mark Sandler
This paper presents an examination of State Space Models (SSM) and Koopman-based deep learning methods for modelling the dynamics of both linear and non-linear stiff strings. Throu…
Narrowing the Focus: Learned Optimizers for Pretrained Models
Gus Kristiansen, Mark Sandler, Andrey Zhmoginov +4
In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand…
Linear Transformers are Versatile In-Context Learners
Max Vladymyrov, Johannes von Oswald, Mark Sandler +1
Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during the…
Uncovering mesa-optimization algorithms in Transformers
Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans +10
Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without be…
The Power of Sparsity in Convolutional Neural Networks
Soravit Changpinyo, Mark Sandler, Andrey Zhmoginov
Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training…