Learning compositional functions via multiplicative weight updates
arXiv:2006.14560
Abstract
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.
References in corpus (13)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Mixed Precision Training
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Training and Inference with Integers in Deep Neural Networks
- Training Deep Neural Networks with 8-bit Floating Point Numbers
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Why gradient clipping accelerates training: A theoretical justification for adaptivity
- A Study of BFLOAT16 for Deep Learning Training
- Rounding Methods for Neural Networks with Low Resolution Synaptic Weights
- On the distance between two neural networks and the stability of learning