Graph-Aware Transformer: Is Attention All Graphs Need?

Abstract

Graphs are the natural data structure to represent relational and structural information in many domains. To cover the broad range of graph-data applications including graph classification as well as graph generation, it is desirable to have a general and flexible model consisting of an encoder and a decoder that can handle graph data. Although the representative encoder-decoder model, Transformer, shows superior performance in various tasks especially of natural language processing, it is not immediately available for graphs due to their non-sequential characteristics. To tackle this incompatibility, we propose GRaph-Aware Transformer (GRAT), the first Transformer-based model which can encode and decode whole graphs in end-to-end fashion. GRAT is featured with a self-attention mechanism adaptive to the edge information and an auto-regressive decoding mechanism based on the two-path approach consisting of sub-graph encoding path and node-and-edge generation path for each decoding step. We empirically evaluated GRAT on multiple setups including encoder-based tasks such as molecule property predictions on QM9 datasets and encoder-decoder-based tasks such as molecule graph generation in the organic molecule synthesis domain. GRAT has shown very promising results including state-of-the-art performance on 4 regression tasks in QM9 benchmark.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Inductive Representation Learning on Large Graphs
• Modeling polypharmacy side effects with graph convolutional networks
• Graph Convolutional Matrix Completion
• Variational Graph Auto-Encoders
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Graph Transformer Networks
• Predicting Organic Reaction Outcomes with Weisfeiler-Lehman Network
• Strategies for Pre-training Graph Neural Networks
• Efficient Graph Generation with Graph Recurrent Attention Networks
• Molecule Attention Transformer
• Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
• Graphite: Iterative Generative Modeling of Graphs
• Multi-Graph Transformer for Free-Hand Sketch Recognition
• Hierarchical Graph-to-Graph Translation for Molecules
• Iterative Visual Reasoning Beyond Convolutions

Cited by in corpus (2)

• Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction
• GeoT: A Geometry-aware Transformer for Reliable Molecular Property Prediction and Chemically Interpretable Representation Learning