Knowledge Consistency between Neural Networks and Beyond

Abstract

This paper aims to analyze knowledge consistency between pre-trained deep neural networks. We propose a generic definition for knowledge consistency between neural networks at different fuzziness levels. A task-agnostic method is designed to disentangle feature components, which represent the consistent knowledge, from raw intermediate-layer features of each neural network. As a generic tool, our method can be broadly used for different applications. In preliminary experiments, we have used knowledge consistency as a tool to diagnose representations of neural networks. Knowledge consistency provides new insights to explain the success of existing deep-learning techniques, such as knowledge distillation and network compression. More crucially, knowledge consistency can also be used to refine pre-trained networks and boost performance.

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• How transferable are features in deep neural networks?
• Understanding Black-box Predictions via Influence Functions
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Opening the Black Box of Deep Neural Networks via Information
• Similarity of Neural Network Representations Revisited
• A Closer Look at Memorization in Deep Networks
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Information-theoretic analysis of generalization capability of learning algorithms
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Representation Learning: A Review and New Perspectives
• Stiffness: A New Perspective on Generalization in Neural Networks
• Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration

Cited by in corpus (9)

• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Grounding Representation Similarity with Statistical Testing
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• Interpreting and Disentangling Feature Components of Various Complexity from DNNs
• ImageNet Pre-training also Transfers Non-Robustness
• Verifiability and Predictability: Interpreting Utilities of Network Architectures for Point Cloud Processing