Grounding Representation Similarity with Statistical Testing

Abstract

To understand neural network behavior, recent works quantitatively compare different networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree on fundamental observations, such as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through a concrete test: measures should have sensitivity to changes that affect functional behavior, and specificity against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail, thus providing a challenge set for further improvement.

arXiv commentAccepted at NeurIPS 2021. 10 pages, 3 figures

References in corpus (15)

• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Sanity Checks for Saliency Maps
• Similarity of Neural Network Representations Revisited
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• What do Neural Machine Translation Models Learn about Morphology?
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
• Weight Poisoning Attacks on Pre-trained Models
• Towards falsifiable interpretability research
• Knowledge Consistency between Neural Networks and Beyond
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Transferred Discrepancy: Quantifying the Difference Between Representations
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level