Boundary between noise and information applied to filtering neural network weight matrices

Abstract

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.

arXiv comment6 pages, 5 figures

References in corpus (17)

• Machine learning and the physical sciences
• Machine learning phases of matter
• Noise Dressing of Financial Correlation Matrices
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Universal and non-universal properties of cross-correlations in financial time series
• Learning phase transitions by confusion
• Fast inference of deep neural networks in FPGAs for particle physics
• Deep Learning and its Application to LHC Physics
• Machine learning quantum phases of matter beyond the fermion sign problem
• Machine Learning Phases of Strongly Correlated Fermions
• Cleaning large correlation matrices: tools from random matrix theory
• Mutual Information, Neural Networks and the Renormalization Group
• Identifying Quantum Phase Transitions with Adversarial Neural Networks
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Physics-informed generative neural network: an application to troposphere temperature prediction
• Random matrix analysis of deep neural network weight matrices

Cited by in corpus (1)

• Task complexity shapes internal representations and robustness in neural networks