Low-Cost Recurrent Neural Network Expected Performance Evaluation

Abstract

Recurrent neural networks are a powerful tool, but they are very sensitive to their hyper-parameter configuration. Moreover, training properly a recurrent neural network is a tough task, therefore selecting an appropriate configuration is critical. Varied strategies have been proposed to tackle this issue. However, most of them are still impractical because of the time/resources needed. In this study, we propose a low computational cost model to evaluate the expected performance of a given architecture based on the distribution of the error of random samples of the weights. We empirically validate our proposal using three use cases. The results suggest that this is a promising alternative to reduce the cost of exploration for hyper-parameter optimization.

References in corpus (2)

• On the difficulty of training Recurrent Neural Networks
• DLOPT: Deep Learning Optimization Library

Cited by in corpus (7)

• Random Error Sampling-based Recurrent Neural Network Architecture Optimization
• Bayesian Neural Architecture Search using A Training-Free Performance Metric
• The Ant Swarm Neuro-Evolution Procedure for Optimizing Recurrent Networks
• DLOPT: Deep Learning Optimization Library
• Reliable and Fast Recurrent Neural Network Architecture Optimization
• An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution
• An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution