On a Sparse Shortcut Topology of Artificial Neural Networks
arXiv:1811.09003
Abstract
In established network architectures, shortcut connections are often used to take the outputs of earlier layers as additional inputs to later layers. Despite the extraordinary effectiveness of shortcuts, there remain open questions on the mechanism and characteristics. For example, why are shortcuts powerful? Why do shortcuts generalize well? In this paper, we investigate the expressivity and generalizability of a novel sparse shortcut topology. First, we demonstrate that this topology can empower a one-neuron-wide deep network to approximate any univariate continuous function. Then, we present a novel width-bounded universal approximator in contrast to depth-bounded universal approximators and extend the approximation result to a family of equally competent networks. Furthermore, with generalization bound theory, we show that the proposed shortcut topology enjoys excellent generalizability. Finally, we corroborate our theoretical analyses by comparing the proposed topology with popular architectures, including ResNet and DenseNet, on well-known benchmarks and perform a saliency map analysis to interpret the proposed topology. Our work helps enhance the understanding of the role of shortcuts and suggests further opportunities to innovate neural architectures.
References in corpus (18)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Axiomatic Attribution for Deep Networks
- Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)
- DARTS: Differentiable Architecture Search
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Training Very Deep Networks
- SmoothGrad: removing noise by adding noise
- On Exact Computation with an Infinitely Wide Neural Net
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Cycle Consistent Adversarial Denoising Network for Multiphase Coronary CT Angiography
- Why Deep Neural Networks for Function Approximation?
- ResNet with one-neuron hidden layers is a Universal Approximator
- The power of deeper networks for expressing natural functions
- Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
- Norm-Based Capacity Control in Neural Networks
- Full-Gradient Representation for Neural Network Visualization
- Towards Understanding the Importance of Shortcut Connections in Residual Networks