All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

Abstract

Deep neural network is difficult to train and this predicament becomes worse as the depth increases. The essence of this problem exists in the magnitude of backpropagated errors that will result in gradient vanishing or exploding phenomenon. We show that a variant of regularizer which utilizes orthonormality among different filter banks can alleviate this problem. Moreover, we design a backward error modulation mechanism based on the quasi-isometry assumption between two consecutive parametric layers. Equipped with these two ingredients, we propose several novel optimization solutions that can be utilized for training a specific-structured (repetitively triple modules of Conv-BNReLU) extremely deep convolutional neural network (CNN) WITHOUT any shortcuts/ identity mappings from scratch. Experiments show that our proposed solutions can achieve distinct improvements for a 44-layer and a 110-layer plain networks on both the CIFAR-10 and ImageNet datasets. Moreover, we can successfully train plain CNNs to match the performance of the residual counterparts. Besides, we propose new principles for designing network structure from the insights evoked by orthonormality. Combined with residual structure, we achieve comparative performance on the ImageNet dataset.

arXiv commentUpdating experiments; CVPR2017

References in corpus (13)

• Deep Residual Learning for Image Recognition
• Wide Residual Networks
• Training Very Deep Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Data-dependent Initializations of Convolutional Neural Networks
• Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
• Random Walk Initialization for Training Very Deep Feedforward Networks
• Learning to Refine Object Segments
• Optimization on Submanifolds of Convolution Kernels in CNNs

Cited by in corpus (19)

• MHSA-Net: Multi-Head Self-Attention Network for Occluded Person Re-Identification
• SVDNet for Pedestrian Retrieval
• Overcoming the vanishing gradient problem in plain recurrent networks
• Deep Hyperspherical Learning
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Unsupervised Meta-Learning for Reinforcement Learning
• Regularizing Activation Distribution for Training Binarized Deep Networks
• On Generalization Bounds of a Family of Recurrent Neural Networks
• MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Information Geometry of Orthogonal Initializations and Training
• Learning towards Minimum Hyperspherical Energy
• Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity
• SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis
• Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy
• Regularizing Neural Networks via Minimizing Hyperspherical Energy
• Homocentric Hypersphere Feature Embedding for Person Re-identification
• GM-Net: Learning Features with More Efficiency
• Filter Bank Regularization of Convolutional Neural Networks