Multi-Residual Networks: Improving the Speed and Accuracy of Residual Networks

Abstract

In this article, we take one step toward understanding the learning behavior of deep residual networks, and supporting the observation that deep residual networks behave like ensembles. We propose a new convolutional neural network architecture which builds upon the success of residual networks by explicitly exploiting the interpretation of very deep networks as an ensemble. The proposed multi-residual network increases the number of residual functions in the residual blocks. Our architecture generates models that are wider, rather than deeper, which significantly improves accuracy. We show that our model achieves an error rate of 3.73% and 19.45% on CIFAR-10 and CIFAR-100 respectively, that outperforms almost all of the existing models. We also demonstrate that our model outperforms very deep residual networks by 0.22% (top-1 error) on the full ImageNet 2012 classification dataset. Additionally, inspired by the parallel structure of multi-residual networks, a model parallelism technique has been investigated. The model parallelism method distributes the computation of residual blocks among the processors, yielding up to 15% computational complexity improvement.

This work has been submitted to the IEEE for possible publication

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• FitNets: Hints for Thin Deep Nets
• Densely Connected Convolutional Networks
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Aggregated Residual Transformations for Deep Neural Networks
• Swapout: Learning an ensemble of deep architectures
• Deep Deconvolutional Networks for Scene Parsing
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks

Cited by in corpus (14)

• The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches
• Shake-Shake regularization
• Selective Kernel Networks
• Deep Convolutional Neural Network Design Patterns
• Light Field Image Quality Assessment With Auxiliary Learning Based on Depthwise and Anglewise Separable Convolutions
• BlockDrop: Dynamic Inference Paths in Residual Networks
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• Artificial Intelligence Assisted Inversion (AIAI) of Synthetic Type Ia Supernova Spectra
• 3D Hand Pose Estimation using Simulation and Partial-Supervision with a Shared Latent Space
• Identity Connections in Residual Nets Improve Noise Stability
• Cost Function Unrolling in Unsupervised Optical Flow
• Face Recognition with Hybrid Efficient Convolution Algorithms on FPGAs
• Comb Convolution for Efficient Convolutional Architecture
• Deep Competitive Pathway Networks