Convergent Learning: Do different neural networks learn the same representations?

Abstract

Recent success in training deep neural networks have prompted active investigation into the features learned on their intermediate layers. Such research is difficult because it requires making sense of non-linear computations performed by millions of parameters, but valuable because it increases our ability to understand current models and create improved versions of them. In this paper we investigate the extent to which neural networks exhibit what we call convergent learning, which is when the representations learned by multiple nets converge to a set of features which are either individually similar between networks or where subsets of features span similar low-dimensional spaces. We propose a specific method of probing representations: training multiple networks and then comparing and contrasting their individual, learned representations at the level of neurons or groups of neurons. We begin research into this question using three techniques to approximately align different neural networks on a feature level: a bipartite matching approach that makes one-to-one assignments between neurons, a sparse prediction approach that finds one-to-many mappings, and a spectral clustering approach that finds many-to-many mappings. This initial investigation reveals a few previously unknown properties of neural networks, and we argue that future research into the question of convergent learning will yield many more. The insights described here include (1) that some features are learned reliably in multiple networks, yet other features are not consistently learned; (2) that units learn to span low-dimensional subspaces and, while these subspaces are common to multiple networks, the specific basis vectors learned are not; (3) that the representation codes show evidence of being a mix between a local code and slightly, but not fully, distributed codes across multiple units.

Published as a conference paper at ICLR 2016

Cited by in corpus (49)

• Shake-Shake regularization
• Edge Preserving and Multi-Scale Contextual Neural Network for Salient Object Detection
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• Revisiting the Importance of Individual Units in CNNs via Ablation
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
• Deep learning as a tool for neural data analysis: speech classification and cross-frequency coupling in human sensorimotor cortex
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles
• A Survey of Deep Learning for Scientific Discovery
• Insights on representational similarity in neural networks with canonical correlation
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• On Interpretability of Artificial Neural Networks: A Survey
• Neural Networks Trained to Solve Differential Equations Learn General Representations
• Understanding trained CNNs by indexing neuron selectivity
• Understanding the Loss Surface of Neural Networks for Binary Classification
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Identifying and Controlling Important Neurons in Neural Machine Translation
• Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
• Emerging Cross-lingual Structure in Pretrained Language Models
• Exploiting Deep Representations for Neural Machine Translation
• Revisiting Model Stitching to Compare Neural Representations
• Transferable Perturbations of Deep Feature Distributions
• Selfish Sparse RNN Training
• Mediators in Determining what Processing BERT Performs First
• Shared Representational Geometry Across Neural Networks
• Statistics of Visual Responses to Object Stimuli from Primate AIT Neurons to DNN Neurons
• Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression
• Model Fusion via Optimal Transport
• PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
• Optimizing Mode Connectivity via Neuron Alignment
• Modeling the Sequential Dependence among Audience Multi-step Conversions with Multi-task Learning in Targeted Display Advertising
• Towards Backward-Compatible Representation Learning
• Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets
• Exploring the Interchangeability of CNN Embedding Spaces
• Deep Network Interpolation for Continuous Imagery Effect Transition
• Topological Insights into Sparse Neural Networks
• RePr: Improved Training of Convolutional Filters
• On the Convergent Properties of Word Embedding Methods
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension
• Scarce Data Driven Deep Learning of Drones via Generalized Data Distribution Space
• Subspace Match Probably Does Not Accurately Assess the Similarity of Learned Representations
• Analyzing Visual Representations in Embodied Navigation Tasks
• Scalable Visual Attribute Extraction through Hidden Layers of a Residual ConvNet
• Efficient Decompositional Rule Extraction for Deep Neural Networks
• Visualizing Classification Structure of Large-Scale Classifiers
• A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
• Robust Disentanglement of a Few Factors at a Time
• Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
• Non-uniqueness phenomenon of object representation in modelling IT cortex by deep convolutional neural network (DCNN)
• Collaborative Group Learning