Graph Structure of Neural Networks

Abstract

Neural networks are often represented as graphs of connections between neurons. However, despite their wide use, there is currently little understanding of the relationship between the graph structure of the neural network and its predictive performance. Here we systematically investigate how does the graph structure of neural networks affect their predictive performance. To this end, we develop a novel graph-based representation of neural networks called relational graph, where layers of neural network computation correspond to rounds of message exchange along the graph structure. Using this representation we show that: (1) a "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance; (2) neural network's performance is approximately a smooth function of the clustering coefficient and average path length of its relational graph; (3) our findings are consistent across many different tasks and datasets; (4) the sweet spot can be identified efficiently; (5) top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks. Our work opens new directions for the design of neural architectures and the understanding on neural networks in general.

ICML 2020, with open-source code

Cited by in corpus (18)

• Design Space for Graph Neural Networks
• Handling Missing Data with Graph Representation Learning
• Automated Machine Learning on Graphs: A Survey
• On Interpretability of Artificial Neural Networks: A Survey
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• sGrapp: Butterfly Approximation in Streaming Graphs
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Clusterability in Neural Networks
• Graph-Based Similarity of Neural Network Representations
• The Elastic Lottery Ticket Hypothesis
• Understanding the Dynamics of DNNs Using Graph Modularity
• Analyzing Neural Networks Based on Random Graphs
• RGP: Neural Network Pruning through Its Regular Graph Structure