Representation Learning on Graphs with Jumping Knowledge Networks

Abstract

Recent deep learning approaches for representation learning on graphs follow a neighborhood aggregation procedure. We analyze some important properties of these models, and propose a strategy to overcome those. In particular, the range of "neighboring" nodes that a node's representation draws from strongly depends on the graph structure, analogous to the spread of a random walk. To adapt to local neighborhood properties and tasks, we explore an architecture -- jumping knowledge (JK) networks -- that flexibly leverages, for each node, different neighborhood ranges to enable better structure-aware representation. In a number of experiments on social, bioinformatics and citation networks, we demonstrate that our model achieves state-of-the-art performance. Furthermore, combining the JK framework with models like Graph Convolutional Networks, GraphSAGE and Graph Attention Networks consistently improves those models' performance.

ICML 2018, accepted as a long oral presentation

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry

Cited by in corpus (249)

• Neural Graph Collaborative Filtering
• KGAT: Knowledge Graph Attention Network for Recommendation
• Graph Neural Networks: A Review of Methods and Applications
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• GNNExplainer: Generating Explanations for Graph Neural Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Graph Neural Networks with convolutional ARMA filters
• Contrastive Multi-View Representation Learning on Graphs
• Scaling Graph Neural Networks with Approximate PageRank
• How Powerful are Graph Neural Networks?
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• A Generalization of Transformer Networks to Graphs
• A Gentle Introduction to Deep Learning for Graphs
• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• Graph Representation Learning in Biomedicine
• DeeperGCN: All You Need to Train Deeper GCNs
• PairNorm: Tackling Oversmoothing in GNNs
• Benchmarking Graph Neural Networks
• Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings
• DeepGCNs: Making GCNs Go as Deep as CNNs
• Combinatorial Optimization with Physics-Inspired Graph Neural Networks
• SumGNN: Multi-typed Drug Interaction Prediction via Efficient Knowledge Graph Summarization
• Towards Sparse Hierarchical Graph Classifiers
• SIGN: Scalable Inception Graph Neural Networks
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Multi-view Graph Contrastive Representation Learning for Drug-Drug Interaction Prediction
• Strategies for Pre-training Graph Neural Networks
• Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs
• Design Space for Graph Neural Networks
• Deep Learning on Graphs: A Survey
• Graph Neural Networks: Taxonomy, Advances and Trends
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• A Survey on Graph-Based Deep Learning for Computational Histopathology
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• GNNGuard: Defending Graph Neural Networks against Adversarial Attacks
• MR-GNN: Multi-Resolution and Dual Graph Neural Network for Predicting Structured Entity Interactions
• Graph Attention Multi-Layer Perceptron
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• Adaptive Propagation Graph Convolutional Network
• Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
• What Can Neural Networks Reason About?
• Graph Information Bottleneck
• Second-Order Pooling for Graph Neural Networks
• Adaptive Universal Generalized PageRank Graph Neural Network
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Attention Guided Graph Convolutional Networks for Relation Extraction
• Representation Learning for Natural Language Processing
• NF-GNN: Network Flow Graph Neural Networks for Malware Detection and Classification
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Learning Dynamic Graph Representation of Brain Connectome with Spatio-Temporal Attention
• Automated Machine Learning on Graphs: A Survey
• MGDCF: Distance Learning via Markov Graph Diffusion for Neural Collaborative Filtering
• A Deep Generative Model for Graph Layout
• Accurate Learning of Graph Representations with Graph Multiset Pooling
• Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Search
• DropMessage: Unifying Random Dropping for Graph Neural Networks
• Semi-Implicit Graph Variational Auto-Encoders
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Graph Neural Ordinary Differential Equations
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
• GraphAIR: Graph Representation Learning with Neighborhood Aggregation and Interaction
• Revisiting Over-smoothing in Deep GCNs
• GIF: A General Graph Unlearning Strategy via Influence Function
• Scalable Graph Neural Networks via Bidirectional Propagation
• Personalized PageRank to a Target Node, Revisited
• Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classification
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Subgraph Neural Networks
• Model Degradation Hinders Deep Graph Neural Networks
• Solving Mixed Integer Programs Using Neural Networks
• Continuous Graph Neural Networks
• AEGCN: An Autoencoder-Constrained Graph Convolutional Network
• Evolutionary Preference Learning via Graph Nested GRU ODE for Session-based Recommendation
• Hypergraph Convolution and Hypergraph Attention
• Intent-aware Recommendation via Disentangled Graph Contrastive Learning
• Multi-hop Attention Graph Neural Network
• Tracklets Predicting Based Adaptive Graph Tracking
• Space4HGNN: A Novel, Modularized and Reproducible Platform to Evaluate Heterogeneous Graph Neural Network
• Understanding over-squashing and bottlenecks on graphs via curvature
• Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
• Graph Star Net for Generalized Multi-Task Learning
• AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models
• BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation
• Edge-Featured Graph Attention Network
• OSLO: On-the-Sphere Learning for Omnidirectional images and its application to 360-degree image compression
• Active Learning for Graph Neural Networks via Node Feature Propagation
• Can Graph Neural Networks Count Substructures?
• Knowledge Graphs and Pre-trained Language Models enhanced Representation Learning for Conversational Recommender Systems
• How does Heterophily Impact the Robustness of Graph Neural Networks? Theoretical Connections and Practical Implications
• New Benchmarks for Learning on Non-Homophilous Graphs
• Lifelong Learning of Graph Neural Networks for Open-World Node Classification
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Bayesian graph convolutional neural networks via tempered MCMC
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• ROD: Reception-aware Online Distillation for Sparse Graphs
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation
• GraphDefense: Towards Robust Graph Convolutional Networks
• Towards a Unified Framework for Fair and Stable Graph Representation Learning
• Probabilistic Dual Network Architecture Search on Graphs
• Pre-train and Learn: Preserve Global Information for Graph Neural Networks
• Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
• Transition Propagation Graph Neural Networks for Temporal Networks
• Beyond Graph Convolutional Network: An Interpretable Regularizer-centered Optimization Framework
• Search to Capture Long-range Dependency with Stacking GNNs for Graph Classification
• Local Augmentation for Graph Neural Networks
• Graph Representation Learning Beyond Node and Homophily
• InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Prediction
• Beyond Graph Neural Networks with Lifted Relational Neural Networks
• Learning to Identify High Betweenness Centrality Nodes from Scratch: A Novel Graph Neural Network Approach
• View-based Explanations for Graph Neural Networks
• Dissecting the Diffusion Process in Linear Graph Convolutional Networks
• Pitfalls in Link Prediction with Graph Neural Networks: Understanding the Impact of Target-link Inclusion & Better Practices
• Graph Neural Networks for User Identity Linkage
• GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks
• Residual Network and Embedding Usage: New Tricks of Node Classification with Graph Convolutional Networks
• Bilinear Graph Neural Network with Neighbor Interactions
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
• Logical Reasoning with Relation Network for Inductive Knowledge Graph Completion
• Towards More Practical Adversarial Attacks on Graph Neural Networks
• TELESTO: A Graph Neural Network Model for Anomaly Classification in Cloud Services
• Node Similarity Preserving Graph Convolutional Networks
• A hybrid deep-learning-metaheuristic framework for bi-level network design problems
• Parameterized Hypercomplex Graph Neural Networks for Graph Classification
• SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks
• Graph Neural Networks Inspired by Classical Iterative Algorithms
• Equivariant Subgraph Aggregation Networks
• Sampling methods for efficient training of graph convolutional networks: A survey
• A Graph Isomorphism Network with Weighted Multiple Aggregators for Speech Emotion Recognition
• NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• Modelling High-Order Social Relations for Item Recommendation
• Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network
• Learning Graph Normalization for Graph Neural Networks
• FedGL: Federated Graph Learning Framework with Global Self-Supervision
• Neural Graph Embedding Methods for Natural Language Processing
• LineMVGNN: Anti-Money Laundering with Line-Graph-Assisted Multi-View Graph Neural Networks
• KECRS: Towards Knowledge-Enriched Conversational Recommendation System
• Fisher-Bures Adversary Graph Convolutional Networks
• Contrastive Graph Neural Network Explanation
• DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction
• On the Unreasonable Effectiveness of Feature propagation in Learning on Graphs with Missing Node Features
• On the Equivalence of Decoupled Graph Convolution Network and Label Propagation
• Graph Neural Bandits
• PPGN: Physics-Preserved Graph Networks for Real-Time Fault Location in Distribution Systems with Limited Observation and Labels
• Dual Policy Learning for Aggregation Optimization in Graph Neural Network-based Recommender Systems
• Power to the Relational Inductive Bias: Graph Neural Networks in Electrical Power Grids
• Adversarial Learned Molecular Graph Inference and Generation
• Directional Graph Networks
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• Zoo Guide to Network Embedding
• SCE: Scalable Network Embedding from Sparsest Cut
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
• Lifelong Graph Learning
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• The Impact of Global Structural Information in Graph Neural Networks Applications
• Mini-batch graphs for robust image classification
• Graph Neural Networks with Parallel Neighborhood Aggregations for Graph Classification
• Information Obfuscation of Graph Neural Networks
• Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs
• Graph Convolutional Networks using Heat Kernel for Semi-supervised Learning
• Siamese Graph Neural Networks for Data Integration
• Graph Node-Feature Convolution for Representation Learning
• Hierarchical Message-Passing Graph Neural Networks
• SNoRe: Scalable Unsupervised Learning of Symbolic Node Representations
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Generating Logical Forms from Graph Representations of Text and Entities
• Black-box Gradient Attack on Graph Neural Networks: Deeper Insights in Graph-based Attack and Defense
• Deepened Graph Auto-Encoders Help Stabilize and Enhance Link Prediction
• Weisfeiler and Lehman Go Paths: Learning Topological Features via Path Complexes
• Deoscillated Graph Collaborative Filtering
• SkipGNN: Predicting Molecular Interactions with Skip-Graph Networks
• Understanding Graph Isomorphism Network for rs-fMRI Functional Connectivity Analysis
• Item Graph Convolution Collaborative Filtering for Inductive Recommendations
• CogDL: A Comprehensive Library for Graph Deep Learning
• Label-Wise Graph Convolutional Network for Heterophilic Graphs
• Weisfeiler and Lehman Go Cellular: CW Networks
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks
• Enabling Automated FPGA Accelerator Optimization Using Graph Neural Networks
• Quaternion Graph Neural Networks
• Accelerating SpMM Kernel with Cache-First Edge Sampling for Graph Neural Networks
• PhysGNN: A Physics-Driven Graph Neural Network Based Model for Predicting Soft Tissue Deformation in Image-Guided Neurosurgery
• Adversarial Graph Disentanglement
• SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks
• $p$-Laplacian Based Graph Neural Networks
• Certified Robustness of Graph Neural Networks against Adversarial Structural Perturbation
• Permutohedral-GCN: Graph Convolutional Networks with Global Attention
• Neural Bandit with Arm Group Graph
• Relation Learning on Social Networks with Multi-Modal Graph Edge Variational Autoencoders
• PushNet: Efficient and Adaptive Neural Message Passing
• Reconstruction for Powerful Graph Representations
• Beyond Homophily with Graph Echo State Networks
• Lossless Compression of Structured Convolutional Models via Lifting
• Scalable Deep Graph Clustering with Random-walk based Self-supervised Learning
• DeepHGCN: Toward Deeper Hyperbolic Graph Convolutional Networks
• Implicit SVD for Graph Representation Learning
• On Self-Distilling Graph Neural Network
• HighwayGraph: Modelling Long-distance Node Relations for Improving General Graph Neural Network
• Breaking the Expressive Bottlenecks of Graph Neural Networks
• HyGCN: A GCN Accelerator with Hybrid Architecture
• Graph Attentional Autoencoder for Anticancer Hyperfood Prediction
• Evaluating Modules in Graph Contrastive Learning
• ENT-DESC: Entity Description Generation by Exploring Knowledge Graph
• Structure-Aware Label Smoothing for Graph Neural Networks
• Graph Neural Networks with Feature and Structure Aware Random Walk
• BiGCN: A Bi-directional Low-Pass Filtering Graph Neural Network
• Locality Preserving Dense Graph Convolutional Networks with Graph Context-Aware Node Representations
• LEReg: Empower Graph Neural Networks with Local Energy Regularization
• Edgeless-GNN: Unsupervised Representation Learning for Edgeless Nodes
• Stacked Graph Filter
• A Hard Label Black-box Adversarial Attack Against Graph Neural Networks
• Embedding Graphs on Grassmann Manifold
• EWasteNet: A Two-Stream Data Efficient Image Transformer Approach for E-Waste Classification
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities
• SAG-VAE: End-to-end Joint Inference of Data Representations and Feature Relations
• Jointly Learnable Data Augmentations for Self-Supervised GNNs
• Graph Filtration Learning
• BiTe-GCN: A New GCN Architecture via BidirectionalConvolution of Topology and Features on Text-Rich Networks
• AKE-GNN: Effective Graph Learning with Adaptive Knowledge Exchange
• Mutual Information Maximization in Graph Neural Networks
• Focus Where It Matters: Graph Selective State Focused Attention Networks
• A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts
• Simplification of Graph Convolutional Networks: A Matrix Factorization-based Perspective
• Learning Conjoint Attentions for Graph Neural Nets
• Regularizing Semi-supervised Graph Convolutional Networks with a Manifold Smoothness Loss
• Exploring Graph Classification Techniques Under Low Data Constraints: A Comprehensive Study
• Layer-stacked Attention for Heterogeneous Network Embedding
• SAS: A Simple, Accurate and Scalable Node Classification Algorithm
• Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation
• Alleviating Over-Smoothing via Aggregation over Compact Manifolds
• Learning to Coordinate via Multiple Graph Neural Networks
• Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs
• Memory-Associated Differential Learning
• Contributions to Representation Learning with Graph Autoencoders and Applications to Music Recommendation
• Dense Graph Convolutional Neural Networks on 3D Meshes for 3D Object Segmentation and Classification
• Densely Connected Graph Convolutional Networks for Graph-to-Sequence Learning
• On the Global Self-attention Mechanism for Graph Convolutional Networks
• Geometric Brain Surface Network For Brain Cortical Parcellation
• DeGNN: Characterizing and Improving Graph Neural Networks with Graph Decomposition
• FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding
• From Cities to Series: Complex Networks and Deep Learning for Improved Spatial and Temporal Analytics*
• Non-Recursive Graph Convolutional Networks