Pregelix: Big(ger) Graph Analytics on A Dataflow Engine

Abstract

There is a growing need for distributed graph processing systems that are capable of gracefully scaling to very large graph datasets. Unfortunately, this challenge has not been easily met due to the intense memory pressure imposed by process-centric, message passing designs that many graph processing systems follow. Pregelix is a new open source distributed graph processing system that is based on an iterative dataflow design that is better tuned to handle both in-memory and out-of-core workloads. As such, Pregelix offers improved performance characteristics and scaling properties over current open source systems (e.g., we have seen up to 15x speedup compared to Apache Giraph and up to 35x speedup compared to distributed GraphLab), and makes more effective use of available machine resources to support Big(ger) Graph Analytics.

References in corpus (1)

• Scaling Datalog for Machine Learning on Big Data

Cited by in corpus (4)

• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Extracting and Analyzing Hidden Graphs from Relational Databases
• DFOGraph: An I/O- and Communication-Efficient System for Distributed Fully-out-of-Core Graph Processing
• Streaming Temporal Graphs: Subgraph Matching