Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
arXiv:2102.00369
Abstract
Useful information (UI) is an elusive concept in neural networks. A quantitative measurement of UI is absent, despite the variations of UI can be recognized by prior knowledge. The communication bandwidth of feature maps decreases after downscaling operations, but UI flows smoothly after training due to lower Nyquist frequency. Inspired by the low-Nyqusit-frequency nature of UI, we propose the use of spectral roll-off points (SROPs) to estimate UI on variations. The computation of an SROP is extended from a 1-D signal to a 2-D image by the required rotation invariance in image classification tasks. SROP statistics across feature maps are implemented as layer-wise useful information estimates. We design sanity checks to explore SROP variations when UI variations are produced by variations in model input, model architecture and training stages. The variations of SROP is synchronizes with UI variations in various randomized and sufficiently trained model structures. Therefore, SROP variations is an accurate and convenient sign of UI variations, which promotes the explainability of data representations with respect to frequency-domain knowledge.
arXiv comment11 pages, 5 figures. This work has been submitted to the IEEE for possible publication
References in corpus (12)
- Understanding Black-box Predictions via Influence Functions
- Opening the Black Box of Deep Neural Networks via Information
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- Estimating the intrinsic dimension of datasets by a minimal neighborhood information
- Making Convolutional Networks Shift-Invariant Again
- A Closer Look at Memorization in Deep Networks
- Exploiting Cyclic Symmetry in Convolutional Neural Networks
- Intrinsic dimension of data representations in deep neural networks
- On the Spectral Bias of Neural Networks
- A Theory of Usable Information Under Computational Constraints
- Knowledge Consistency between Neural Networks and Beyond
- Towards Task and Architecture-Independent Generalization Gap Predictors