78 citations · 191 across the 19 of their papers we have counts for
16 papers · 1 filter
SOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models
Pit Neitemeier, Alessio Serra, Jiaze Li +3
Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end…
u-P: The Unit-Scaled Maximal Update Parametrization
Charlie Blake, Constantin Eichenberg, Josef Dean +7
The Maximal Update Parametrization (P) aims to make the optimal hyperparameters (HPs) of a model independent of its size, allowing them to be swept using a cheap proxy model rat…
Choice of PEFT Technique in Continual Learning: Prompt Tuning is Not All You Need
Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles +1
Recent Continual Learning (CL) methods have combined pretrained Transformers with prompt tuning, a parameter-efficient fine-tuning (PEFT) technique. We argue that the choice of pro…
A Negative Result on Gradient Matching for Selective Backprop
Lukas Balles, Cedric Archambeau, Giovanni Zappella
With increasing scale in model and dataset size, the training of deep neural networks becomes a massive computational burden. One approach to speed up the training process is Selec…
Continual Learning with Low Rank Adaptation
Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles +1
Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest. However, they struggle to retain that…
Renate: A Library for Real-World Continual Learning
Martin Wistuba, Martin Ferianc, Lukas Balles +2
Continual learning enables the incremental training of machine learning models on non-stationary data streams.While academic interest in the topic is high, there is little indicati…
PASHA: Efficient HPO and NAS with Progressive Resource Allocation
Ondrej Bohdal, Lukas Balles, Martin Wistuba +3
Hyperparameter optimization (HPO) and neural architecture search (NAS) are methods of choice to obtain the best-in-class machine learning models, but in practice they can be costly…
Gradient-Matching Coresets for Rehearsal-Based Continual Learning
Lukas Balles, Giovanni Zappella, Cédric Archambeau
The goal of continual learning (CL) is to efficiently update a machine learning model with new data without forgetting previously-learned knowledge. Most widely-used CL methods rel…
Gradient-matching coresets for continual learning
Lukas Balles, Giovanni Zappella, Cédric Archambeau
We devise a coreset selection method based on the idea of gradient matching: The gradients induced by the coreset should match, as closely as possible, those induced by the origina…
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering
Ricky T. Q. Chen, Dami Choi, Lukas Balles +2
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such…
The Geometry of Sign Gradient Descent
Lukas Balles, Fabian Pedregosa, Nicolas Le Roux
Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performanc…
Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
Frederik Kunstner, Lukas Balles, Philipp Hennig
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-o…
DeepOBS: A Deep Learning Optimizer Benchmark Suite
Frank Schneider, Lukas Balles, Philipp Hennig
Because the choice and tuning of the optimizer affects the speed, and ultimately the performance of deep learning, there is significant past and recent research in this area. Yet,…
Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
Lukas Balles, Philipp Hennig
The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: fo…
Early Stopping without a Validation Set
Maren Mahsereci, Lukas Balles, Christoph Lassner +1
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good…
Coupling Adaptive Batch Sizes with Learning Rates
Lukas Balles, Javier Romero, Philipp Hennig
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are…