Understanding the Dynamics of DNNs Using Graph Modularity

Abstract

There are good arguments to support the claim that deep neural networks (DNNs) capture better feature representations than the previous hand-crafted feature engineering, which leads to a significant performance improvement. In this paper, we move a tiny step towards understanding the dynamics of feature representations over layers. Specifically, we model the process of class separation of intermediate representations in pre-trained DNNs as the evolution of communities in dynamic graphs. Then, we introduce modularity, a generic metric in graph theory, to quantify the evolution of communities. In the preliminary experiment, we find that modularity roughly tends to increase as the layer goes deeper and the degradation and plateau arise when the model complexity is great relative to the dataset. Through an asymptotic analysis, we prove that modularity can be broadly used for different applications. For example, modularity provides new insights to quantify the difference between feature representations. More crucially, we demonstrate that the degradation and plateau in modularity curves represent redundant layers in DNNs and can be pruned with minimal impact on performance, which provides theoretical guidance for layer pruning. Our code is available at https://github.com/yaolu-zjut/Dynamic-Graphs-Construction.

Accepted by ECCV 2022

References in corpus (13)

• Sequence to Sequence Learning with Neural Networks
• Modularity and community structure in networks
• Cooperative Game Theory Approaches for Network Partitioning
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Similarity of Neural Network Representations Revisited
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Graph Structure of Neural Networks
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Clusterability in Neural Networks
• Probing neural networks with t-SNE, class-specific projections and a guided tour