CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks

Abstract

Despite the recent advances of graph neural networks (GNNs) in modeling graph data, the training of GNNs on large datasets is notoriously hard due to the overfitting. Adversarial training, which augments data with the worst-case adversarial examples, has been widely demonstrated to improve model's robustness against adversarial attacks and generalization ability. However, while the previous adversarial training generally focuses on protecting GNNs from spiteful attacks, it remains unclear how the adversarial training could improve the generalization abilities of GNNs in the graph analytics problem. In this paper, we investigate GNNs from the lens of weight and feature loss landscapes, i.e., the loss changes with respect to model weights and node features, respectively. We draw the conclusion that GNNs are prone to falling into sharp local minima in these two loss landscapes, where GNNs possess poor generalization performances. To tackle this problem, we construct the co-adversarial perturbation (CAP) optimization problem in terms of weights and features, and design the alternating adversarial perturbation algorithm to flatten the weight and feature loss landscapes alternately. Furthermore, we divide the training process into two stages: one conducting the standard cross-entropy minimization to ensure the quick convergence of GNN models, the other applying our alternating adversarial training to avoid falling into locally sharp minima. The extensive experiments demonstrate our CAP can generally improve the generalization performance of GNNs on a variety of benchmark graph datasets.

References in corpus (16)

• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Convolutional Networks on Graphs for Learning Molecular Fingerprints
• DeepInf: Social Influence Prediction with Deep Learning
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Adversarial Training Can Hurt Generalization
• Sharp Minima Can Generalize For Deep Nets
• Geom-GCN: Geometric Graph Convolutional Networks
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Improving Neural Language Modeling via Adversarial Training
• Batch Virtual Adversarial Training for Graph Convolutional Networks
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Adversarial Feature Augmentation and Normalization for Visual Recognition

Cited by in corpus (1)

• Causal invariant geographic network representations with feature and structural distribution shifts