430 citations · 607 across the 19 of their papers we have counts for
3 papers · 1 filter
Learning and Unlearning of Fabricated Knowledge in Language Models
Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov +2
What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this…
Narrowing the Focus: Learned Optimizers for Pretrained Models
Gus Kristiansen, Mark Sandler, Andrey Zhmoginov +4
In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand…
Linear Transformers are Versatile In-Context Learners
Max Vladymyrov, Johannes von Oswald, Mark Sandler +1
Recent research has demonstrated that transformers, particularly linear attention models, implicitly execute gradient-descent-like algorithms on data provided in-context during the…