Tandem Blocks in Deep Convolutional Neural Networks
arXiv:1806.00145
Abstract
Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers. We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small image classification networks. Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts. Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.
arXiv comment10 pages, 6 figures
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- FractalNet: Ultra-Deep Neural Networks without Residuals
- Highway Networks
- Residual Networks of Residual Networks: Multilevel Residual Networks
- Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
- Highway and Residual Networks learn Unrolled Iterative Estimation
- PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
- Deep Pyramidal Residual Networks
- Wide-Residual-Inception Networks for Real-time Object Detection
- Learning Identity Mappings with Residual Gates