DiracNets: Training Very Deep Neural Networks Without Skip-Connections

Abstract

Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip-connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution-ReLU pairs. We are able to match ResNet-1001 accuracy on CIFAR-10 with 28-layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non-residual networks. The code and models for our experiments are available at https://github.com/szagoruyko/diracnets

References in corpus (5)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Densely Connected Convolutional Networks
• Deeply-Supervised Nets
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Return of the Devil in the Details: Delving Deep into Convolutional Nets

Cited by in corpus (9)

• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• RepVGG: Making VGG-style ConvNets Great Again
• EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks
• Deep Learning for Image Denoising: A Survey
• Bayesian regression explains how human participants handle parameter uncertainty
• Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks