Function Norms and Regularization in Deep Networks
arXiv:1710.06703
Abstract
Deep neural networks (DNNs) have become increasingly important due to their excellent empirical performance on a wide range of problems. However, regularization is generally achieved by indirect means, largely due to the complex set of functions defined by a network and the difficulty in measuring function complexity. There exists no method in the literature for additive regularization based on a norm of the function, as is classically considered in statistical learning theory. In this work, we propose sampling-based approximations to weighted function norms as regularizers for deep neural networks. We provide, to the best of our knowledge, the first proof in the literature of the NP-hardness of computing function norms of DNNs, motivating the necessity of an approximate approach. We then derive a generalization bound for functions trained with weighted norms and prove that a natural stochastic optimization strategy minimizes the bound. Finally, we empirically validate the improved performance of the proposed regularization strategies for both convex function sets as well as DNNs on real-world classification and image segmentation tasks demonstrating improved performance over weight decay, dropout, and batch normalization. Source code will be released at the time of publication.
arXiv comment17 pages, 8 figures
References in corpus (6)
- Auto-Encoding Variational Bayes
- Improving neural networks by preventing co-adaptation of feature detectors
- ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Norm-Based Capacity Control in Neural Networks