Heterogeneous Multilayer Generalized Operational Perceptron

Abstract

The traditional Multilayer Perceptron (MLP) using McCulloch-Pitts neuron model is inherently limited to a set of neuronal activities, i.e., linear weighted sum followed by nonlinear thresholding step. Previously, Generalized Operational Perceptron (GOP) was proposed to extend conventional perceptron model by defining a diverse set of neuronal activities to imitate a generalized model of biological neurons. Together with GOP, Progressive Operational Perceptron (POP) algorithm was proposed to optimize a pre-defined template of multiple homogeneous layers in a layerwise manner. In this paper, we propose an efficient algorithm to learn a compact, fully heterogeneous multilayer network that allows each individual neuron, regardless of the layer, to have distinct characteristics. Based on the complexity of the problem, the proposed algorithm operates in a progressive manner on a neuronal level, searching for a compact topology, not only in terms of depth but also width, i.e., the number of neurons in each layer. The proposed algorithm is shown to outperform other related learning methods in extensive experiments on several classification problems.

arXiv commentAccepted in IEEE Transaction on Neural Networks and Learning Systems

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Wide Residual Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Progressive Learning for Systematic Design of Large Neural Networks
• Layer-wise training of deep networks using kernel similarity

Cited by in corpus (10)

• Data-driven fluid mechanics of wind farms: A review
• Robust Peak Detection for Holter ECGs by Self-Organized Operational Neural Networks
• 1-Dimensional polynomial neural networks for audio signal related problems
• Progressive Operational Perceptron with Memory
• Data-driven Neural Architecture Learning For Financial Time-series Forecasting
• Operational vs Convolutional Neural Networks for Image Denoising
• Neural Architecture Search by Estimation of Network Structure Distributions
• Self-Organized Variational Autoencoders (Self-VAE) for Learned Image Compression
• Operational Support Estimator Networks
• Knowledge Distillation By Sparse Representation Matching