8 papers
Towards flexible perception with visual memory
Robert Geirhos, Priyank Jaini, Austin Stone +5
Training a neural network is a monolithic endeavor, akin to carving knowledge into stone: once the process is completed, editing the knowledge in a network is hard, since all infor…
Benchmarking Neural Network Training Algorithms
George E. Dahl, Frank Schneider, Zachary Nado +22
Training algorithms, broadly construed, are an essential part of every deep learning pipeline. Training algorithm improvements that speed up training across a wide variety of workl…
How far away are truly hyperparameter-free learning algorithms?
Priya Kasimbeg, Vincent Roulet, Naman Agarwal +4
Despite major advances in methodology, hyperparameter tuning remains a crucial (and expensive) part of the development of machine learning systems. Even ignoring architectural choi…
Learning by solving differential equations
Benoit Dherin, Michael Munn, Hanna Mazzawi +3
Modern deep learning algorithms use variations of gradient descent as their main learning methods. Gradient descent can be understood as the simplest Ordinary Differential Equation…
Training neural networks faster with minimal tuning using pre-computed lists of hyperparameters for NAdamW
Sourabh Medapati, Priya Kasimbeg, Shankar Krishnan +2
If we want to train a neural network using any of the most popular optimization algorithms, we are immediately faced with a dilemma: how to set the various optimization and regular…
Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition
Priya Kasimbeg, Frank Schneider, Runa Eschenhagen +11
The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorit…