Graph-adaptive Rectified Linear Unit for Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. The key to the GNNs is adopting the neural message-passing paradigm with two stages: aggregation and update. The current design of GNNs considers the topology information in the aggregation stage. However, in the updating stage, all nodes share the same updating function. The identical updating function treats each node embedding as i.i.d. random variables and thus ignores the implicit relationships between neighborhoods, which limits the capacity of the GNNs. The updating function is usually implemented with a linear transformation followed by a non-linear activation function. To make the updating function topology-aware, we inject the topological information into the non-linear activation function and propose Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. The parameters of GReLU are obtained from a hyperfunction based on both node features and the corresponding adjacent matrix. To reduce the risk of overfitting and the computational cost, we decompose the hyperfunction as two independent components for nodes and features respectively. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.

arXiv commentTheWebConf (WWW), 2022

References in corpus (15)

• A Comprehensive Survey on Graph Neural Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Simplifying Graph Convolutional Networks
• MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding
• Graph Neural Networks with convolutional ARMA filters
• Predict then Propagate: Graph Neural Networks meet Personalized PageRank
• STP-UDGAT: Spatial-Temporal-Preference User Dimensional Graph Attention Network for Next POI Recommendation
• How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
• Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space
• Power Normalizations in Fine-grained Image, Few-shot Image and Graph Classification
• Graph-based Semi-supervised Learning: A Comprehensive Review
• Contrastive Laplacian Eigenmaps
• REFINE: Random RangE FInder for Network Embedding
• Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation
• Dynamic ReLU

Cited by in corpus (4)

• COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning
• HICF: Hyperbolic Informative Collaborative Filtering
• Because Every Sensor Is Unique, so Is Every Pair: Handling Dynamicity in Traffic Forecasting
• Graph Self-Supervised Learning with Learnable Structural and Positional Encodings