23 citations · 25 across the 4 of their papers we have counts for
4 papers
Learning and Unlearning of Fabricated Knowledge in Language Models
Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov +2
What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this…
Narrowing the Focus: Learned Optimizers for Pretrained Models
Gus Kristiansen, Mark Sandler, Andrey Zhmoginov +4
In modern deep learning, the models are learned by applying gradient updates using an optimizer, which transforms the updates based on various statistics. Optimizers are often hand…
Training trajectories, mini-batch losses and the curious role of the learning rate
Mark Sandler, Andrey Zhmoginov, Max Vladymyrov +1
Stochastic gradient descent plays a fundamental role in nearly all applications of deep learning. However its ability to converge to a global minimum remains shrouded in mystery. I…
Partial-Hessian Strategies for Fast Learning of Nonlinear Embeddings
Max Vladymyrov, Miguel Carreira-Perpinan
Stochastic neighbor embedding (SNE) and related nonlinear manifold learning algorithms achieve high-quality low-dimensional representations of similarity data, but are notoriously…