Understanding the Dynamics of DNNs Using Graph Modularity

Abstract

There are good arguments to support the claim that deep neural networks (DNNs) capture better feature representations than the previous hand-crafted feature engineering, which leads to a significant performance improvement. In this paper, we move a tiny step towards understanding the dynamics of feature representations over layers. Specifically, we model the process of class separation of intermediate representations in pre-trained DNNs as the evolution of communities in dynamic graphs. Then, we introduce modularity, a generic metric in graph theory, to quantify the evolution of communities. In the preliminary experiment, we find that modularity roughly tends to increase as the layer goes deeper and the degradation and plateau arise when the model complexity is great relative to the dataset. Through an asymptotic analysis, we prove that modularity can be broadly used for different applications. For example, modularity provides new insights to quantify the difference between feature representations. More crucially, we demonstrate that the degradation and plateau in modularity curves represent redundant layers in DNNs and can be pruned with minimal impact on performance, which provides theoretical guidance for layer pruning. Our code is available at https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.

arXiv commentAccepted by ECCV 2022

References in corpus (23)

• Finding and evaluating community structure in networks
• Sequence to Sequence Learning with Neural Networks
• Modularity and community structure in networks
• Community detection in graphs
• Cooperative Game Theory Approaches for Network Partitioning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• A continual learning survey: Defying forgetting in classification tasks
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Similarity of Neural Network Representations Revisited
• Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Modular Networks: Learning to Decompose Neural Computation
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
• Graph Structure of Neural Networks
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• Similarity of Neural Networks with Gradients
• On Network Science and Mutual Information for Explaining Deep Neural Networks
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Clusterability in Neural Networks
• Interpreting Layered Neural Networks via Hierarchical Modular Representation
• Probing neural networks with t-SNE, class-specific projections and a guided tour