Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
arXiv:1612.06519
Abstract
In recent years, the research community has discovered that deep neural networks (DNNs) and convolutional neural networks (CNNs) can yield higher accuracy than all previous solutions to a broad array of machine learning problems. To our knowledge, there is no single CNN/DNN architecture that solves all problems optimally. Instead, the "right" CNN/DNN architecture varies depending on the application at hand. CNN/DNNs comprise an enormous design space. Quantitatively, we find that a small region of the CNN design space contains 30 billion different CNN architectures. In this dissertation, we develop a methodology that enables systematic exploration of the design space of CNNs. Our methodology is comprised of the following four themes. 1. Judiciously choosing benchmarks and metrics. 2. Rapidly training CNN models. 3. Defining and describing the CNN design space. 4. Exploring the design space of CNN architectures. Taken together, these four themes comprise an effective methodology for discovering the "right" CNN architectures to meet the needs of practical applications.
arXiv commentthesis, UC Berkeley (2016)
References in corpus (40)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Distributed Representations of Words and Phrases and their Compositionality
- Practical Bayesian Optimization of Machine Learning Algorithms
- Deep Residual Learning for Image Recognition
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Going Deeper with Convolutions
- End-to-End Training of Deep Visuomotor Policies
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- cuDNN: Efficient Primitives for Deep Learning
- One weird trick for parallelizing convolutional neural networks
- DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Deep Image: Scaling up Image Recognition
- Recent Advances in Convolutional Neural Networks
- Systematic evaluation of CNN advances on the ImageNet
- Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
- Origami: A 803 GOp/s/W Convolutional Network Accelerator
- Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
- DSD: Dense-Sparse-Dense Training for Deep Neural Networks
- Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
- SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving
- SparkNet: Training Deep Networks in Spark
- DeepLogo: Hitting Logo Recognition with the Deep Neural Network Hammer
- How to scale distributed deep learning?
- The Effects of Hyperparameters on SGD Training of Neural Networks
- Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
- Theano-based Large-Scale Visual Recognition with Multiple GPUs
- maxDNN: An Efficient Convolution Kernel for Deep Learning with Maxwell GPUs
- Theano-MPI: a Theano-based Distributed Training Framework
- Purine: A bi-graph based deep learning framework
- A Metaprogramming and Autotuning Framework for Deploying Deep Learning Applications
- Performance of GTX Titan X GPUs and Code Optimization