PyTorch-BigGraph: A Large-scale Graph Embedding System

Abstract

Graph embedding methods produce unsupervised node features from graphs that can then be used for a variety of machine learning tasks. Modern graphs, particularly in industrial applications, contain billions of nodes and trillions of edges, which exceeds the capability of existing embedding systems. We present PyTorch-BigGraph (PBG), an embedding system that incorporates several modifications to traditional multi-relation embedding systems that allow it to scale to graphs with billions of nodes and trillions of edges. PBG uses graph partitioning to train arbitrarily large embeddings on either a single machine or in a distributed environment. We demonstrate comparable performance with existing embedding systems on common benchmarks, while allowing for scaling to arbitrarily large graphs and parallelization on multiple machines. We train and evaluate embeddings on several large social network graphs as well as the full Freebase dataset, which contains over 100 million nodes and 2 billion edges.

References in corpus (9)

• Efficient Estimation of Word Representations in Vector Space
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• A Review of Relational Machine Learning for Knowledge Graphs
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Predicting multicellular function through multi-layer tissue networks
• Type-Constrained Representation Learning in Knowledge Graphs
• Swivel: Improving Embeddings by Noticing What's Missing
• MILE: A Multi-Level Framework for Scalable Graph Embedding

Cited by in corpus (47)

• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Automated Machine Learning on Graphs: A Survey
• DGL-KE: Training Knowledge Graph Embeddings at Scale
• Modern Baselines for SPARQL Semantic Parsing
• AGL: a Scalable System for Industrial-purpose Graph Machine Learning
• Scalable Graph Neural Networks for Heterogeneous Graphs
• MILE: A Multi-Level Framework for Scalable Graph Embedding
• PANE: scalable and effective attributed network embedding
• Dynamic Parameter Allocation in Parameter Servers
• COIN: Communication-Aware In-Memory Acceleration for Graph Convolutional Networks
• Taming Momentum in a Distributed Asynchronous Environment
• NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs
• Knowledge Graph Embeddings and Explainable AI
• Marius: Learning Massive Graph Embeddings on a Single Machine
• A Distributed Multi-GPU System for Large-Scale Node Embedding at Tencent
• DeepTrax: Embedding Graphs of Financial Transactions
• Consolidating Commonsense Knowledge
• TinyKG: Memory-Efficient Training Framework for Knowledge Graph Neural Recommender Systems
• FastGL: A GPU-Efficient Framework for Accelerating Sampling-Based GNN Training at Large Scale
• Source-Aware Embedding Training on Heterogeneous Information Networks
• On the Ambiguity of Rank-Based Evaluation of Entity Alignment or Link Prediction Methods
• Homogeneous Network Embedding for Massive Graphs via Reweighted Personalized PageRank
• Russo-Ukrainian War: Prediction and explanation of Twitter suspension
• Parallel Computation of Graph Embeddings
• Leiden-Fusion Partitioning Method for Effective Distributed Training of Graph Embeddings
• WikiDataSets: Standardized sub-graphs from Wikidata
• FairMILE: Towards an Efficient Framework for Fair Graph Representation Learning
• Social World Knowledge: Modeling and Applications
• GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs
• Growing and Serving Large Open-domain Knowledge Graphs
• Towards Lightweight and Automated Representation Learning System for Networks
• HUGE: Huge Unsupervised Graph Embeddings with TPUs
• AdaGNN: A multi-modal latent representation meta-learner for GNNs based on AdaBoosting
• Enriching language models with graph-based context information to better understand textual data
• UniNet: Scalable Network Representation Learning with Metropolis-Hastings Sampling
• MOOCRep: A Unified Pre-trained Embedding of MOOC Entities
• Scalable Graph Embedding LearningOn A Single GPU
• Bot-Match: Social Bot Detection with Recursive Nearest Neighbors Search
• kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval
• Good Intentions: Adaptive Parameter Management via Intent Signaling
• TreeGAN: Incorporating Class Hierarchy into Image Generation
• TIES: Temporal Interaction Embeddings For Enhancing Social Media Integrity At Facebook
• Prioritizing Original News on Facebook
• Rating and aspect-based opinion graph embeddings for explainable recommendations
• Graphing else matters: exploiting aspect opinions and ratings in explainable graph-based recommendations
• ScaleFreeCTR: MixCache-based Distributed Training System for CTR Models with Huge Embedding Table