Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardware
arXiv:1901.09181
Abstract
Artificial Neural Networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very-high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as Graphics Processing Unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity inducing techniques either operate from a pre-trained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high dimensional data, while achieving higher accuracy than the traditional two phases approaches. Moreover, we have been able to create truly sparse MultiLayer Perceptrons (MLPs) models with over one million neurons and to train them on a typical laptop without GPU, this being way beyond what is possible with any state-of-the-art techniques.
16 pages
References in corpus (16)
- Improving neural networks by preventing co-adaptation of feature detectors
- Reconciling modern machine learning practice and the bias-variance trade-off
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
- Variational Dropout Sparsifies Deep Neural Networks
- Sparse Networks from Scratch: Faster Training without Losing Performance
- Lifelong Learning with Dynamically Expandable Networks
- Deep Rewiring: Training very sparse deep networks
- Exploring Sparsity in Recurrent Neural Networks
- Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
- Learning Intrinsic Sparse Structures within Long Short-Term Memory
- A topological insight into restricted Boltzmann machines
- Estimating 3D Trajectories from 2D Projections via Disjunctive Factored Four-Way Conditional Restricted Boltzmann Machines
- On improving deep learning generalization with adaptive sparse connectivity
- Intrinsically Sparse Long Short-Term Memory Networks