Neuralizing Efficient Higher-order Belief Propagation

Abstract

Graph neural network models have been extensively used to learn node representations for graph structured data in an end-to-end setting. These models often rely on localized first order approximations of spectral graph convolutions and hence are unable to capture higher-order relational information between nodes. Probabilistic Graphical Models form another class of models that provide rich flexibility in incorporating such relational information but are limited by inefficient approximate inference algorithms at higher order. In this paper, we propose to combine these approaches to learn better node and graph representations. First, we derive an efficient approximate sum-product loopy belief propagation inference algorithm for higher-order PGMs. We then embed the message passing updates into a neural network to provide the inductive bias of the inference algorithm in end-to-end learning. This gives us a model that is flexible enough to accommodate domain knowledge while maintaining the computational advantage. We further propose methods for constructing higher-order factors that are conditioned on node and edge features and share parameters wherever necessary. Our experimental evaluation shows that our model indeed captures higher-order information, substantially outperforming state-of-the-art $k$-order graph neural networks in molecular datasets.

References in corpus (11)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Fast Graph Representation Learning with PyTorch Geometric
• Order Matters: Sequence to sequence for sets
• Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models
• Hyper-SAGNN: a self-attention based graph neural network for hypergraphs
• Deeply Learning the Messages in Message Passing Inference
• Factor Graph Neural Network