How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks
arXiv:2010.05983
Abstract
This paper studies the novel concept of weight correlation in deep neural networks and discusses its impact on the networks' generalisation ability. For fully-connected layers, the weight correlation is defined as the average cosine similarity between weight vectors of neurons, and for convolutional layers, the weight correlation is defined as the cosine similarity between filter matrices. Theoretically, we show that, weight correlation can, and should, be incorporated into the PAC Bayesian framework for the generalisation of neural networks, and the resulting generalisation bound is monotonic with respect to the weight correlation. We formulate a new complexity measure, which lifts the PAC Bayes measure with weight correlation, and experimentally confirm that it is able to rank the generalisation errors of a set of networks more precisely than existing measures. More importantly, we develop a new regulariser for training, and provide extensive experiments that show that the generalisation error can be greatly reduced with our novel approach.
Accpeted by NeurIPS 2020 conference
References in corpus (10)
- A Convergence Theory for Deep Learning via Over-Parameterization
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Norm-Based Capacity Control in Neural Networks
- A non-parametric k-nearest neighbour entropy estimator
- Generalization in Deep Networks: The Role of Distance from Initialization
- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
- Generalization bounds for deep convolutional neural networks