SSFN -- Self Size-estimating Feed-forward Network with Low Complexity, Limited Need for Human Intervention, and Consistent Behaviour across Trials

Abstract

We design a self size-estimating feed-forward network (SSFN) using a joint optimization approach for estimation of number of layers, number of nodes and learning of weight matrices. The learning algorithm has a low computational complexity, preferably within few minutes using a laptop. In addition the algorithm has a limited need for human intervention to tune parameters. SSFN grows from a small-size network to a large-size network, guaranteeing a monotonically non-increasing cost with addition of nodes and layers. The learning approach uses judicious a combination of `lossless flow property' of some activation functions, convex optimization and instance of random matrix. Consistent performance -- low variation across Monte-Carlo trials -- is found for inference performance (classification accuracy) and estimation of network size.

References in corpus (10)

• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• AutoAugment: Learning Augmentation Policies from Data
• Learned Convolutional Sparse Coding
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Analysis of Regularized LS Reconstruction and Random Matrix Ensembles in Compressed Sensing
• Forward Thinking: Building and Training Neural Networks One Layer at a Time
• Progressive Learning for Systematic Design of Large Neural Networks
• Layer-wise training of deep networks using kernel similarity
• Bayesian Learning of Neural Network Architectures

Cited by in corpus (5)

• Predictive Analysis of COVID-19 Time-series Data from Johns Hopkins University
• Asynchronous Decentralized Learning of a Neural Network
• Neural Greedy Pursuit for Feature Selection
• High-dimensional Neural Feature Design for Layer-wise Reduction of Training Cost
• Statistical model-based evaluation of neural networks