Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks

Abstract

Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks, where now excessively large models are used. However, such models face several problems during the learning process, mainly due to the redundancy of the individual neurons, which results in sub-optimal accuracy or the need for additional training steps. Here, we explore the diversity of the neurons within the hidden layer during the learning process, and analyze how the diversity of the neurons affects predictions of the model. As following, we introduce several techniques to dynamically reinforce diversity between neurons during the training. These decorrelation techniques improve learning at early stages and occasionally help to overcome local minima faster. Additionally, we describe novel weight initialization method to obtain decorrelated, yet stochastic weight initialization for a fast and efficient neural network training. Decorrelated weight initialization in our case shows about 40% relative increase in test accuracy during the first 5 epochs.

arXiv commentAccepted to ICANN 2021

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Language Models are Few-Shot Learners
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Collapse of Deep and Narrow Neural Nets
• Dying ReLU and Initialization: Theory and Numerical Examples
• The Expressive Power of Neural Networks: A View from the Width
• When does Diversity Help Generalization in Classification Ensembles?
• Local minima in training of neural networks
• Inverse Problems, Deep Learning, and Symmetry Breaking
• Symmetry & critical points for a model shallow neural network