Understanding Generalization in Deep Learning via Tensor Methods

Abstract

Deep neural networks generalize well on unseen data though the number of parameters often far exceeds the number of training examples. Recently proposed complexity measures have provided insights to understanding the generalizability in neural networks from perspectives of PAC-Bayes, robustness, overparametrization, compression and so on. In this work, we advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. Using tensor analysis, we propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks; thus, in practice, our generalization bound outperforms the previous compression-based ones, especially for neural networks using tensors as their weight kernels (e.g. CNNs). Moreover, these intuitive measurements provide further insights into designing neural network architectures with properties favorable for better/guaranteed generalizability. Our experimental results demonstrate that through the proposed measurable properties, our generalization error bound matches the trend of the test error well. Our theoretical analysis further provides justifications for the empirical success and limitations of some widely-used tensor-based compression approaches. We also discover the improvements to the compressibility and robustness of current neural networks when incorporating tensor operations via our proposed layer-wise structure.

arXiv comment9 pages (main paper), 42 pages (full version)

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• TensorLy: Tensor Learning in Python
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• The Singular Values of Convolutional Layers
• Ultimate tensorization: compressing convolutional and FC layers alike
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Norm-Based Capacity Control in Neural Networks
• Generalization Error of Invariant Classifiers
• Fast Rates for Empirical Risk Minimization of Strict Saddle Problems
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Guaranteed Simultaneous Asymmetric Tensor Decomposition via Orthogonalized Alternating Least Squares

Cited by in corpus (7)

• Tensor Methods in Computer Vision and Deep Learning
• On generalization bounds for deep networks based on loss surface implicit regularization
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Using Wavelets and Spectral Methods to Study Patterns in Image-Classification Datasets
• Approximate Contraction of Arbitrary Tensor Networks with a Flexible and Efficient Density Matrix Algorithm
• Implicit Regularization in Tensor Factorization
• Using Wavelets to Analyze Similarities in Image-Classification Datasets