Reinforcement Learning for Learning Rate Control

Abstract

Stochastic gradient descent (SGD), which updates the model parameters by adding a local gradient times a learning rate at each step, is widely used in model training of machine learning algorithms such as neural networks. It is observed that the models trained by SGD are sensitive to learning rates and good learning rates are problem specific. We propose an algorithm to automatically learn learning rates using neural network based actor-critic methods from deep reinforcement learning (RL).In particular, we train a policy network called actor to decide the learning rate at each step during training, and a value network called critic to give feedback about quality of the decision (e.g., the goodness of the learning rate outputted by the actor) that the actor made. The introduction of auxiliary actor and critic networks helps the main network achieve better performance. Experiments on different datasets and network architectures show that our approach leads to better convergence of SGD than human-designed competitors.

7 pages, 9 figures

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Gradient-based Hyperparameter Optimization through Reversible Learning
• No More Pesky Learning Rates
• An Actor-Critic Algorithm for Sequence Prediction

Cited by in corpus (8)

• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• A Generalizable Approach to Learning Optimizers
• Gradient-only line searches to automatically determine learning rates for a variety of stochastic training algorithms
• Traversing the noise of dynamic mini-batch sub-sampled loss functions: A visual guide
• OLR-WA: Online Weighted Average Linear Regression in Multivariate Data Streams
• MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server