Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network

Abstract

One of the biggest issues in deep learning theory is the generalization ability of networks with huge model size. The classical learning theory suggests that overparameterized models cause overfitting. However, practically used large deep models avoid overfitting, which is not well explained by the classical approaches. To resolve this issue, several attempts have been made. Among them, the compression based bound is one of the promising approaches. However, the compression based bound can be applied only to a compressed network, and it is not applicable to the non-compressed original network. In this paper, we give a unified frame-work that can convert compression based bounds to those for non-compressed original networks. The bound gives even better rate than the one for the compressed network by improving the bias term. By establishing the unified frame-work, we can obtain a data dependent generalization error bound which gives a tighter evaluation than the data independent ones.

arXiv commentpublished in ICLR2020

References in corpus (20)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Understanding deep learning requires rethinking generalization
• On the Number of Linear Regions of Deep Neural Networks
• Manifold Mixup: Better Representations by Interpolating Hidden States
• 2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Dropout Training as Adaptive Regularization
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Concentration inequalities and asymptotic results for ratio type empirical processes
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Gradient descent aligns the layers of deep linear networks
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Norm-Based Capacity Control in Neural Networks
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Cited by in corpus (4)

• On generalization bounds for deep networks based on loss surface implicit regularization
• Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD