Analysis and Optimization of Convolutional Neural Network Architectures

Abstract

Convolutional Neural Networks (CNNs) dominate various computer vision tasks since Alex Krizhevsky showed that they can be trained effectively and reduced the top-5 error from 26.2 % to 15.3 % on the ImageNet large scale visual recognition challenge. Many aspects of CNNs are examined in various publications, but literature about the analysis and construction of neural network architectures is rare. This work is one step to close this gap. A comprehensive overview over existing techniques for CNN analysis and topology construction is provided. A novel way to visualize classification errors with confusion matrices was developed. Based on this method, hierarchical classifiers are described and evaluated. Additionally, some results are confirmed and quantified for CIFAR-100. For example, the positive impact of smaller batch sizes, averaging ensembles, data augmentation and test-time transformations on the accuracy. Other results, such as the positive impact of learned color transformation on the test accuracy could not be confirmed. A model which has only one million learned parameters for an input size of 32x32x3 and 100 classes and which beats the state of the art on the benchmark dataset Asirra, GTSRB, HASYv2 and STL-10 was developed.

Master's thesis. 73 pages + 24 pages appendix; 39 figures; 33 tables

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Intriguing properties of neural networks
• ADADELTA: An Adaptive Learning Rate Method
• An overview of gradient descent optimization algorithms
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding deep learning requires rethinking generalization
• cuDNN: Efficient Primitives for Deep Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Fully Convolutional Networks for Semantic Segmentation
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Aggregated Residual Transformations for Deep Neural Networks
• Deep Image: Scaling up Image Recognition
• Systematic evaluation of CNN advances on the ImageNet
• What makes ImageNet good for transfer learning?
• Deep Networks with Stochastic Depth
• Stacked What-Where Auto-encoders
• All you need is a good init
• Places: An Image Database for Deep Scene Understanding
• Convolutional Neural Fabrics
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• On orthogonality and learning recurrent networks with long term dependencies
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• A Survey of Semantic Segmentation
• Generative NeuroEvolution for Deep Learning
• Towards Competitive Classifiers for Unbalanced Classification Problems: A Study on the Performance Scores

Cited by in corpus (6)

• On Class Orderings for Incremental Learning
• Faster Neural Network Training with Approximate Tensor Operations
• Asymptotic Singular Value Distribution of Linear Convolutional Layers
• WSAM: Visual Explanations from Style Augmentation as Adversarial Attacker and Their Influence in Image Classification
• Fast Efficient Object Detection Using Selective Attention
• Faster Convolution Inference Through Using Pre-Calculated Lookup Tables