Watch Your Step: Learning Node Embeddings via Graph Attention

Abstract

Graph embedding methods represent nodes in a continuous vector space, preserving information from the graph (e.g. by sampling random walks). There are many hyper-parameters to these methods (such as random walk length) which have to be manually tuned for every graph. In this paper, we replace random walk hyper-parameters with trainable parameters that we automatically learn via backpropagation. In particular, we learn a novel attention model on the power series of the transition matrix, which guides the random walk to optimize an upstream objective. Unlike previous approaches to attention models, the method that we propose utilizes attention parameters exclusively on the data (e.g. on the random walk), and not used by the model for inference. We experiment on link prediction tasks, as we aim to produce embeddings that best-preserve the graph structure, generalizing to unseen information. We improve state-of-the-art on a comprehensive suite of real world datasets including social, collaboration, and biological networks. Adding attention to random walks can reduce the error by 20% to 45% on datasets we attempted. Further, our learned attention parameters are different for every graph, and our automatically-found values agree with the optimal choice of hyper-parameter if we manually tune existing methods.

Cited by in corpus (42)

• A Comprehensive Survey on Graph Neural Networks
• Graph Learning: A Survey
• An Overview of Deep Semi-Supervised Learning
• Differentiable Graph Module (DGM) for Graph Convolutional Networks
• Is a Single Embedding Enough? Learning Node Representations that Capture Multiple Social Contexts
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Adaptive Universal Generalized PageRank Graph Neural Network
• DDGK: Learning Graph Representations for Deep Divergence Graph Kernels
• A Tutorial on Network Embeddings
• Path Integral Based Convolution and Pooling for Graph Neural Networks
• Benchmarking Network Embedding Models for Link Prediction: Are We Making Progress?
• HAHE: Hierarchical Attentive Heterogeneous Information Network Embedding
• Improving Graph Attention Networks with Large Margin-based Constraints
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
• Global Attention Improves Graph Networks Generalization
• IsoNN: Isomorphic Neural Network for Graph Representation Learning and Classification
• Link prediction in dynamic networks using random dot product graphs
• AWB-GCN: A Graph Convolutional Network Accelerator with Runtime Workload Rebalancing
• DeepTrax: Embedding Graphs of Financial Transactions
• Network Representation Learning: Consolidation and Renewed Bearing
• Multiple Kernel Representation Learning on Networks
• DAGCN: Dual Attention Graph Convolutional Networks
• Feature Propagation on Graph: A New Perspective to Graph Representation Learning
• Semi-Supervised Learning on Graphs Based on Local Label Distributions
• Self-Supervised Learning of Contextual Embeddings for Link Prediction in Heterogeneous Networks
• Homogeneous Network Embedding for Massive Graphs via Reweighted Personalized PageRank
• $p$-Laplacian Based Graph Neural Networks
• PINE: Universal Deep Embedding for Graph Nodes via Partial Permutation Invariant Set Functions
• Implicit SVD for Graph Representation Learning
• Understanding Higher-order Structures in Evolving Graphs: A Simplicial Complex based Kernel Estimation Approach
• Towards Lightweight and Automated Representation Learning System for Networks
• Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learning
• Graph Embedding VAE: A Permutation Invariant Model of Graph Structure
• Graph Convolutional Networks Meet with High Dimensionality Reduction
• Temporal Network Representation Learning via Historical Neighborhoods Aggregation
• Graph Representation Learning Network via Adaptive Sampling
• Automated Feature-Topic Pairing: Aligning Semantic and Embedding Spaces in Spatial Representation Learning
• Learning Embeddings of Directed Networks with Text-Associated Nodes---with Applications in Software Package Dependency Networks
• Fast Graph Learning with Unique Optimal Solutions
• A Literature Review of Recent Graph Embedding Techniques for Biomedical Data
• Dynamic Embedding on Textual Networks via a Gaussian Process
• Learning Based Proximity Matrix Factorization for Node Embedding