GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding

Abstract

Learning continuous representations of nodes is attracting growing interest in both academia and industry recently, due to their simplicity and effectiveness in a variety of applications. Most of existing node embedding algorithms and systems are capable of processing networks with hundreds of thousands or a few millions of nodes. However, how to scale them to networks that have tens of millions or even hundreds of millions of nodes remains a challenging problem. In this paper, we propose GraphVite, a high-performance CPU-GPU hybrid system for training node embeddings, by co-optimizing the algorithm and the system. On the CPU end, augmented edge samples are parallelly generated by random walks in an online fashion on the network, and serve as the training data. On the GPU end, a novel parallel negative sampling is proposed to leverage multiple GPUs to train node embeddings simultaneously, without much data transfer and synchronization. Moreover, an efficient collaboration strategy is proposed to further reduce the synchronization cost between CPUs and GPUs. Experiments on multiple real-world networks show that GraphVite is super efficient. It takes only about one minute for a network with 1 million nodes and 5 million edges on a single machine with 4 GPUs, and takes around 20 hours for a network with 66 million nodes and 1.8 billion edges. Compared to the current fastest system, GraphVite is about 50 times faster without any sacrifice on performance.

accepted at WWW 2019

References in corpus (17)

• Distributed Representations of Words and Phrases and their Compositionality
• DeepWalk: Online Learning of Social Representations
• TensorFlow: A system for large-scale machine learning
• Semi-Supervised Classification with Graph Convolutional Networks
• Playing Atari with Deep Reinforcement Learning
• LINE: Large-scale Information Network Embedding
• Neural Message Passing for Quantum Chemistry
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Mixed Precision Training
• Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec
• Visualizing Large-scale and High-dimensional Data
• VERSE: Versatile Graph Embeddings from Similarity Measures
• Deep Graph Infomax
• NOMAD: Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completion
• Parallelizing Word2Vec in Multi-Core and Many-Core Architectures
• BMF: Block matrix approach to factorization of large scale data

Cited by in corpus (21)

• Graph Neural Networks: A Review of Methods and Applications
• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• Trustworthy Graph Neural Networks: Aspects, Methods and Trends
• Knowledge Graph informed Fake News Classification via Heterogeneous Representation Ensembles
• PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• PANE: scalable and effective attributed network embedding
• A Retrieve-and-Read Framework for Knowledge Graph Link Prediction
• SketchNE: Embedding Billion-Scale Networks Accurately in One Hour
• Marius: Learning Massive Graph Embeddings on a Single Machine
• A Distributed Multi-GPU System for Large-Scale Node Embedding at Tencent
• TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
• FairMILE: Towards an Efficient Framework for Fair Graph Representation Learning
• Homogeneous Network Embedding for Massive Graphs via Reweighted Personalized PageRank
• Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings
• Towards Lightweight and Automated Representation Learning System for Networks
• HUGE: Huge Unsupervised Graph Embeddings with TPUs
• KGTN-ens: Few-Shot Image Classification with Knowledge Graph Ensembles
• Graph4Rec: A Universal Toolkit with Graph Neural Networks for Recommender Systems
• Scalable Graph Embedding LearningOn A Single GPU
• CitationIE: Leveraging the Citation Graph for Scientific Information Extraction