SpinalNet: Deep Neural Network with Gradual Input

Abstract

Deep neural networks (DNNs) have achieved the state of the art performance in numerous fields. However, DNNs need high computation times, and people always expect better performance in a lower computation. Therefore, we study the human somatosensory system and design a neural network (SpinalNet) to achieve higher accuracy with fewer computations. Hidden layers in traditional NNs receive inputs in the previous layer, apply activation function, and then transfer the outcomes to the next layer. In the proposed SpinalNet, each layer is split into three splits: 1) input split, 2) intermediate split, and 3) output split. Input split of each layer receives a part of the inputs. The intermediate split of each layer receives outputs of the intermediate split of the previous layer and outputs of the input split of the current layer. The number of incoming weights becomes significantly lower than traditional DNNs. The SpinalNet can also be used as the fully connected or classification layer of DNN and supports both traditional learning and transfer learning. We observe significant error reductions with lower computational costs in most of the DNNs. Traditional learning on the VGG-5 network with SpinalNet classification layers provided the state-of-the-art (SOTA) performance on QMNIST, Kuzushiji-MNIST, EMNIST (Letters, Digits, and Balanced) datasets. Traditional learning with ImageNet pre-trained initial weights and SpinalNet classification layers provided the SOTA performance on STL-10, Fruits 360, Bird225, and Caltech-101 datasets. The scripts of the proposed SpinalNet are available at the following link: https://github.com/dipuk0506/SpinalNet

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Wide Residual Networks
• Mathematical Formulation of Multi-Layer Networks
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• A Unified approach for Conventional Zero-shot, Generalized Zero-shot and Few-shot Learning
• Neural Architecture Transfer
• CINIC-10 is not ImageNet or CIFAR-10
• ResNet with one-neuron hidden layers is a Universal Approximator
• Training Neural Networks with Local Error Signals
• RMDL: Random Multimodel Deep Learning for Classification
• AdaNet: Adaptive Structural Learning of Artificial Neural Networks
• FMix: Enhancing Mixed Sample Data Augmentation
• Domain Adaptive Transfer Learning with Specialist Models
• Cold Case: The Lost MNIST Digits
• TextCaps : Handwritten Character Recognition with Very Small Datasets
• Classification of Shoulder X-Ray Images with Deep Learning Ensemble Models
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Morphological Classification of Galaxies Using SpinalNet
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Context-Aware Multipath Networks
• Self-Evolutionary Clustering

Cited by in corpus (11)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Review of Generalized Zero-Shot Learning Methods
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• A comparative study of cosmological constraints from weak lensing using Convolutional Neural Networks
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality
• Deep Dimension Reduction for Supervised Representation Learning
• WSAM: Visual Explanations from Style Augmentation as Adversarial Attacker and Their Influence in Image Classification
• Sequential Random Network for Fine-grained Image Classification
• Conceptual Expansion Neural Architecture Search (CENAS)
• Signature-Graph Networks
• ProgressiveSpinalNet architecture for FC layers