Learning by Turning: Neural Architecture Aware Optimisation
arXiv:2102.07227
Abstract
Descent methods for deep networks are notoriously capricious: they require careful tuning of step size, momentum and weight decay, and which method will work best on a new benchmark is a priori unclear. To address this problem, this paper conducts a combined study of neural architecture and optimisation, leading to a new optimiser called Nero: the neuronal rotator. Nero trains reliably without momentum or weight decay, works in situations where Adam and SGD fail, and requires little to no learning rate tuning. Also, Nero's memory footprint is ~ square root that of Adam or LAMB. Nero combines two ideas: (1) projected gradient descent over the space of balanced networks; (2) neuron-specific updates, where the step size sets the angle through which each neuron's hyperplane turns. The paper concludes by discussing how this geometric connection between architecture and optimisation may impact theories of generalisation in deep learning.
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- On the Convergence of Adam and Beyond
- Optimization for deep learning: theory and algorithms
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- Deep learning generalizes because the parameter-function map is biased towards simple functions
- ProxQuant: Quantized Neural Networks via Proximal Operators
- On the distance between two neural networks and the stability of learning
- Minnorm training: an algorithm for training over-parameterized deep neural networks
- Orthogonal Over-Parameterized Training