Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
arXiv:1901.11458
Abstract
Deep neural networks are over-parameterized, which implies that the number of parameters are much larger than the number of samples used to train the network. Even in such a regime deep architectures do not overfit. This phenomenon is an active area of research and many theories have been proposed trying to understand this peculiar observation. These include the Vapnik Chervonenkis (VC) dimension bounds and Rademacher complexity bounds which show that the capacity of the network is characterized by the norm of weights rather than the number of parameters. However, the effect of input noise on these measures for shallow and deep architectures has not been studied. In this paper, we analyze the effects of various regularization schemes on the complexity of a neural network which we characterize with the loss, norm of the weights, Rademacher complexities (Directly Approximately Regularizing Complexity-DARC1), VC dimension based Low Complexity Neural Network (LCNN) when subject to varying degrees of Gaussian input noise. We show that regularization leads to a simpler hypothesis class and better generalization followed by DARC1 regularizer, both for shallow as well as deeper architectures. Jacobian regularizer works well for shallow architectures with high level of input noises. Spectral normalization attains highest test set accuracies both for shallow and deeper architectures. We also show that Dropout alone does not perform well in presence of input noise. Finally, we show that deeper architectures are robust to input noise as opposed to their shallow counterparts.
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- LSTM: A Search Space Odyssey
- Spectral Normalization for Generative Adversarial Networks
- Wide Residual Networks
- Understanding deep learning requires rethinking generalization
- VQA: Visual Question Answering
- Shake-Shake regularization
- Robust Large Margin Deep Neural Networks
- ShakeDrop Regularization for Deep Residual Learning
- Stronger generalization bounds for deep nets via a compression approach
- Learning Neural Network Classifiers with Low Model Complexity
- Radius-margin bounds for deep neural networks