Residual Networks of Residual Networks: Multilevel Residual Networks

Abstract

A residual-networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability. This paper proposes a novel residual-network architecture, Residual networks of Residual networks (RoR), to dig the optimization ability of residual networks. RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping. In particular, RoR adds level-wise shortcut connections upon original residual networks to promote the learning capability of residual networks. More importantly, RoR can be applied to various kinds of residual networks (ResNets, Pre-ResNets and WRN) and significantly boost their performance. Our experiments demonstrate the effectiveness and versatility of RoR, where it achieves the best performance in all residual-network-like structures. Our RoR-3-WRN58-4+SD models achieve new state-of-the-art results on CIFAR-10, CIFAR-100 and SVHN, with test errors 3.77%, 19.73% and 1.59%, respectively. RoR-3 models also achieve state-of-the-art results compared to ResNets on ImageNet data set.

arXiv commentIEEE Transactions on Circuits and Systems for Video Technology 2017

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improving neural networks by preventing co-adaptation of feature detectors
• Striving for Simplicity: The All Convolutional Net
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• FitNets: Hints for Thin Deep Nets
• Wide Residual Networks
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Deeply-Supervised Nets
• Resnet in Resnet: Generalizing Residual Architectures
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Deep Networks with Stochastic Depth
• Identity Mappings in Deep Residual Networks
• All you need is a good init
• Swapout: Learning an ensemble of deep architectures
• Deep Residual Networks with Exponential Linear Unit
• Dark Forces in the Sky: Signals from Z' and the Dark Higgs
• Convolutional Residual Memory Networks

Cited by in corpus (31)

• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Recent Advances in Convolutional Neural Networks
• Fast convolutional neural networks on FPGAs with hls4ml
• Wavelet Convolutional Neural Networks
• Selective Kernel Networks
• On the Origin of Deep Learning
• Deep Polynomial Neural Networks
• Harnessing Multi-View Perspective of Light Fields for Low-Light Imaging
• Deep Convolutional Neural Network Design Patterns
• PolyNet: A Pursuit of Structural Diversity in Very Deep Networks
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• Learning for Disparity Estimation through Feature Constancy
• Fine-Grained Age Estimation in the wild with Attention LSTM Networks
• Age Group and Gender Estimation in the Wild with Deep RoR Architecture
• A Robust Deep Networks based Multi-Object MultiCamera Tracking System for City Scale Traffic
• Improved Stereo Matching with Constant Highway Networks and Reflective Confidence Learning
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• A Framework for 3D Tracking of Frontal Dynamic Objects in Autonomous Cars
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Improving training of deep neural networks via Singular Value Bounding
• Building a Regular Decision Boundary with Deep Networks
• AFN: Attentional Feedback Network based 3D Terrain Super-Resolution
• MU-GAN: Facial Attribute Editing based on Multi-attention Mechanism
• Tandem Blocks in Deep Convolutional Neural Networks
• ActiveMatch: End-to-end Semi-supervised Active Representation Learning
• 4-Connected Shift Residual Networks
• Reconciling Feature-Reuse and Overfitting in DenseNet with Specialized Dropout
• Mixture separability loss in a deep convolutional network for image classification
• Short-term Load Forecasting with Deep Residual Networks
• Competing Ratio Loss for Discriminative Multi-class Image Classification