Similarity of Neural Network Models: A Survey of Functional and Representational Measures

Abstract

Measuring similarity of neural networks to understand and improve their behavior has become an issue of great importance and research interest. In this survey, we provide a comprehensive overview of two complementary perspectives of measuring neural network similarity: (i) representational similarity, which considers how activations of intermediate layers differ, and (ii) functional similarity, which considers how models differ in their outputs. In addition to providing detailed descriptions of existing measures, we summarize and discuss results on the properties of and relationships between these measures, and point to open research problems. We hope our work lays a foundation for more systematic research on the properties and applicability of similarity measures for neural network models.

ACM Computing Surveys

References in corpus (20)

• Modularity and community structure in networks
• Measuring and testing dependence by correlation of distances
• Understanding metric-related pitfalls in image analysis validation
• Do Vision Transformers See Like Convolutional Neural Networks?
• Getting aligned on representational alignment
• On the Reproducibility of Neural Network Predictions
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Evaluating Explainability for Graph Neural Networks
• A Survey on Self-Supervised Representation Learning
• Anti-Distillation: Improving reproducibility of deep networks
• On the Origins of the Block Structure Phenomenon in Neural Network Representations
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Exploring the Interchangeability of CNN Embedding Spaces
• Geometric and Topological Inference for Deep Representations of Complex Networks
• Undivided Attention: Are Intermediate Layers Necessary for BERT?
• Graph-Based Similarity of Neural Network Representations
• Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency
• Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamics
• SimSMoE: Solving Representational Collapse via Similarity Measure
• Understanding Weight Similarity of Neural Networks via Chain Normalization Rule and Hypothesis-Training-Testing

Cited by in corpus (2)

• FedCTTA: A Collaborative Approach to Continual Test-Time Adaptation in Federated Learning
• Task complexity shapes internal representations and robustness in neural networks