Progressive Learning for Systematic Design of Large Neural Networks

Abstract

We develop an algorithm for systematic design of a large artificial neural network using a progression property. We find that some non-linear functions, such as the rectifier linear unit and its derivatives, hold the property. The systematic design addresses the choice of network size and regularization of parameters. The number of nodes and layers in network increases in progression with the objective of consistently reducing an appropriate cost. Each layer is optimized at a time, where appropriate parameters are learned using convex optimization. Regularization parameters for convex optimization do not need a significant manual effort for tuning. We also use random instances for some weight matrices, and that helps to reduce the number of parameters we learn. The developed network is expected to show good generalization power due to appropriate regularization and use of random weights in the layers. This expectation is verified by extensive experiments for classification and regression problems, using standard databases.

Cited by in corpus (7)

• Heterogeneous Multilayer Generalized Operational Perceptron
• Progressive Operational Perceptron with Memory
• SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials
• Data-driven Neural Architecture Learning For Financial Time-series Forecasting
• High-dimensional Neural Feature Design for Layer-wise Reduction of Training Cost
• A Low Complexity Decentralized Neural Net with Centralized Equivalence using Layer-wise Learning
• R3Net: Random Weights, Rectifier Linear Units and Robustness for Artificial Neural Network