Tune smarter not harder: A principled approach to tuning learning rates for shallow nets

Abstract

Effective hyper-parameter tuning is essential to guarantee the performance that neural networks have come to be known for. In this work, a principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. We associate the learning rate with the gradient Lipschitz constant of the objective to be minimized while training. An upper bound on the mentioned constant is derived and a search algorithm, which always results in non-divergent traces, is proposed to exploit the derived bound. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods such as Tree Parzen Estimators (TPE). The proposed method is applied to three different existing applications: a) channel estimation in OFDM systems, b) prediction of the exchange currency rates and c) offset estimation in OFDM receivers, and it is shown to pick better learning rates than the existing methods using the same or lesser compute power.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• A Survey on Deep Learning in Medical Image Analysis
• Practical Bayesian Optimization of Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• Learning Optimal Resource Allocations in Wireless Systems
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Lipschitz regularity of deep neural networks: analysis and efficient estimation
• Learning One-hidden-layer ReLU Networks via Gradient Descent
• Structured and Unstructured Outlier Identification for Robust PCA: A Non iterative, Parameter free Algorithm
• Lipschitz Properties for Deep Convolutional Networks
• Signal and Noise Statistics Oblivious Orthogonal Matching Pursuit