Deep Convolutional Neural Networks with Merge-and-Run Mappings

Abstract

A deep residual network, built by stacking a sequence of residual blocks, is easy to train, because identity mappings skip residual branches and thus improve information flow. To further reduce the training difficulty, we present a simple network architecture, deep merge-and-run neural networks. The novelty lies in a modularized building block, merge-and-run block, which assembles residual branches in parallel through a merge-and-run mapping: Average the inputs of these residual branches (Merge), and add the average to the output of each residual branch as the input of the subsequent residual branch (Run), respectively. We show that the merge-and-run mapping is a linear idempotent function in which the transformation matrix is idempotent, and thus improves information flow, making training easy. In comparison to residual networks, our networks enjoy compelling advantages: they contain much shorter paths, and the width, i.e., the number of channels, is increased. We evaluate the performance on the standard recognition tasks. Our approach demonstrates consistent improvements over ResNets with the comparable setup, and achieves competitive results (e.g., $3.57\%$ testing error on CIFAR-$10$, $19.00\%$ on CIFAR-$100$, $1.51\%$ on SVHN).

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• FitNets: Hints for Thin Deep Nets
• Densely Connected Convolutional Networks
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Fully Convolutional Networks for Semantic Segmentation
• Resnet in Resnet: Generalizing Residual Architectures
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Residual Networks of Residual Networks: Multilevel Residual Networks
• Swapout: Learning an ensemble of deep architectures
• Deeply-Fused Nets
• Deep Convolutional Neural Network Design Patterns

Cited by in corpus (12)

• COVID-19 Chest CT Image Segmentation -- A Deep Convolutional Neural Network Solution
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks
• Optimization Algorithm Inspired Deep Neural Network Structure Design
• Accelerating Deep Neural Networks with Spatial Bottleneck Modules
• Rethink ReLU to Training Better CNNs
• WaveletNet: Logarithmic Scale Efficient Convolutional Neural Networks for Edge Devices
• Challenge report:VIPriors Action Recognition Challenge
• Fully Automated and Standardized Segmentation of Adipose Tissue Compartments by Deep Learning in Three-dimensional Whole-body MRI of Epidemiological Cohort Studies
• Short-term Load Forecasting with Deep Residual Networks
• Self-grouping Convolutional Neural Networks