GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Abstract

Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.

arXiv commentICML 2021, Code: https://github.com/lsj2408/GraphNorm

References in corpus (29)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Spectral Networks and Locally Connected Networks on Graphs
• Graph Neural Networks: A Review of Methods and Applications
• Representation Learning on Graphs with Jumping Knowledge Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• DeeperGCN: All You Need to Train Deeper GCNs
• Benchmarking Graph Neural Networks
• PairNorm: Tackling Oversmoothing in GNNs
• Do Transformers Really Perform Bad for Graph Representation?
• Universal Invariant and Equivariant Graph Neural Networks
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• Anonymous Walk Embeddings
• Norm matters: efficient and accurate normalization schemes in deep networks
• What Can Neural Networks Reason About?
• Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• Revisiting Over-smoothing in Deep GCNs
• MoleculeNet: A Benchmark for Molecular Machine Learning
• An Exponential Learning Rate Schedule for Deep Learning
• Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• How hard is to distinguish graphs with graph neural networks?
• Learning Graph Normalization for Graph Neural Networks

Cited by in corpus (16)

• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Vision Transformers with Patch Diversification
• An end-to-end attention-based approach for learning on graphs
• AnchorGAE: General Data Clustering via $O(n)$ Bipartite Graph Convolution
• Information Obfuscation of Graph Neural Networks
• Graph convolutions that can finally model local structure
• Wasserstein Embedding for Graph Learning
• Heterogeneous Directed Hypergraph Neural Network over abstract syntax tree (AST) for Code Classification
• On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction
• Convergent Graph Solvers
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities
• Latent Graph Representations for Critical View of Safety Assessment
• Equivariant Graph Attention Networks with Structural Motifs for Predicting Cell Line-Specific Synergistic Drug Combinations
• Group Whitening: Balancing Learning Efficiency and Representational Capacity