Improving Graph Attention Networks with Large Margin-based Constraints

Abstract

Graph Attention Networks (GATs) are the state-of-the-art neural architecture for representation learning with graphs. GATs learn attention functions that assign weights to nodes so that different nodes have different influences in the feature aggregation steps. In practice, however, induced attention functions are prone to over-fitting due to the increasing number of parameters and the lack of direct supervision on attention weights. GATs also suffer from over-smoothing at the decision boundary of nodes. Here we propose a framework to address their weaknesses via margin-based constraints on attention during training. We first theoretically demonstrate the over-smoothing behavior of GATs and then develop an approach using constraint on the attention weights according to the class boundary and feature aggregation pattern. Furthermore, to alleviate the over-fitting problem, we propose additional constraints on the graph structure. Extensive experiments and ablation studies on common benchmark datasets demonstrate the effectiveness of our method, which leads to significant improvements over the previous state-of-the-art graph attention methods on all datasets.

References in corpus (17)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Simplifying Graph Convolutional Networks
• Graph Convolutional Matrix Completion
• Learning Convolutional Neural Networks for Graphs
• Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Attention-based Graph Neural Network for Semi-supervised Learning
• Adaptive Sampling Towards Fast Graph Representation Learning
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• Watch Your Step: Learning Node Embeddings via Graph Attention
• Deeply learning molecular structure-property relationships using attention- and gate-augmented graph convolutional network
• GeniePath: Graph Neural Networks with Adaptive Receptive Paths
• Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs
• Graph Neural Networks and Boolean Satisfiability

Cited by in corpus (12)

• Principal Neighbourhood Aggregation for Graph Nets
• How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• How Attentive are Graph Attention Networks?
• Vision Transformers with Patch Diversification
• Multi-hop Attention Graph Neural Network
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• ProGraML: Graph-based Deep Learning for Program Optimization and Analysis
• Pointer Graph Networks
• Node Masking: Making Graph Neural Networks Generalize and Scale Better
• Deepened Graph Auto-Encoders Help Stabilize and Enhance Link Prediction
• Learning Conjoint Attentions for Graph Neural Nets