Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
arXiv:1801.07650
Abstract
Deep neural networks (DNNs) are powerful machine learning models and have succeeded in various artificial intelligence tasks. Although various architectures and modules for the DNNs have been proposed, selecting and designing the appropriate network structure for a target problem is a challenging task. In this paper, we propose a method to simultaneously optimize the network structure and weight parameters during neural network training. We consider a probability distribution that generates network structures, and optimize the parameters of the distribution instead of directly optimizing the network structure. The proposed method can apply to the various network structure optimization problems under the same framework. We apply the proposed method to several structure optimization problems such as selection of layers, selection of unit types, and selection of connections using the MNIST, CIFAR-10, and CIFAR-100 datasets. The experimental results show that the proposed method can find the appropriate and competitive network structures.
To appear in the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 9 pages
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Learning Structured Sparsity in Deep Neural Networks
- Deep Networks with Stochastic Depth
- Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
- Swapout: Learning an ensemble of deep architectures
- A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
Cited by in corpus (3)
- You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient
- Learning sparse transformations through backpropagation
- Parameterless Stochastic Natural Gradient Method for Discrete Optimization and its Application to Hyper-Parameter Optimization for Neural Network