Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities

Abstract

Graph neural networks (GNNs) have been widely used to learn vector representation of graph-structured data and achieved better task performance than conventional methods. The foundation of GNNs is the message passing procedure, which propagates the information in a node to its neighbors. Since this procedure proceeds one step per layer, the range of the information propagation among nodes is small in the lower layers, and it expands toward the higher layers. Therefore, a GNN model has to be deep enough to capture global structural information in a graph. On the other hand, it is known that deep GNN models suffer from performance degradation because they lose nodes' local information, which would be essential for good model performance, through many message passing steps. In this study, we propose multi-level attention pooling (MLAP) for graph-level classification tasks, which can adapt to both local and global structural information in a graph. It has an attention pooling layer for each message passing step and computes the final graph representation by unifying the layer-wise graph representations. The MLAP architecture allows models to utilize the structural information of graphs with multiple levels of localities because it preserves layer-wise information before losing them due to oversmoothing. Results of our experiments show that the MLAP architecture improves the graph classification performance compared to the baseline architectures. In addition, analyses on the layer-wise graph representations suggest that aggregating information from multiple levels of localities indeed has the potential to improve the discriminability of learned graph representations.

arXiv comment17 pages + 5 appendix pages, 16 figures + 2 appendix figures, 2 tables + 5 appendix tables

References in corpus (26)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Machine Translation by Jointly Learning to Align and Translate
• A Comprehensive Survey on Graph Neural Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Spectral Networks and Locally Connected Networks on Graphs
• KGAT: Knowledge Graph Attention Network for Recommendation
• Representation Learning on Graphs with Jumping Knowledge Networks
• Learning Convolutional Neural Networks for Graphs
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Layer Normalization
• Simple and Deep Graph Convolutional Networks
• TUDataset: A collection of benchmark datasets for learning with graphs
• PairNorm: Tackling Oversmoothing in GNNs
• Towards Sparse Hierarchical Graph Classifiers
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Deep Learning on Graphs: A Survey
• Fractality in complex networks: critical and supercritical skeletons
• Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Scattering GCN: Overcoming Oversmoothness in Graph Convolutional Networks
• GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks