75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices

Abstract

The SuiteSparse GraphBLAS C-library implements high performance hypersparse matrices with bindings to a variety of languages (Python, Julia, and Matlab/Octave). GraphBLAS provides a lightweight in-memory database implementation of hypersparse matrices that are ideal for analyzing many types of network data, while providing rigorous mathematical guarantees, such as linearity. Streaming updates of hypersparse matrices put enormous pressure on the memory hierarchy. This work benchmarks an implementation of hierarchical hypersparse matrices that reduces memory pressure and dramatically increases the update rate into a hypersparse matrices. The parameters of hierarchical hypersparse matrices rely on controlling the number of entries in each level in the hierarchy before an update is cascaded. The parameters are easily tunable to achieve optimal performance for a variety of applications. Hierarchical hypersparse matrices achieve over 1,000,000 updates per second in a single instance. Scaling to 31,000 instances of hierarchical hypersparse matrices arrays on 1,100 server nodes on the MIT SuperCloud achieved a sustained update rate of 75,000,000,000 updates per second. This capability allows the MIT SuperCloud to analyze extremely large streaming network data sets.

4 pages, 4 figures, 28 references, accepted to IPDPS GrAPL 2020. arXiv admin note: substantial text overlap with arXiv:1907.04217

References in corpus (8)

• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• Mathematical Foundations of the GraphBLAS
• Static Graph Challenge: Subgraph Isomorphism
• Streaming Graph Challenge: Stochastic Block Partition
• Achieving 100,000,000 database inserts per second using Accumulo and D4M
• Hypersparse Neural Network Analysis of Large-Scale Internet Traffic
• Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M
• Benchmarking SciDB Data Import on HPC Systems

Cited by in corpus (7)

• Spatial Temporal Analysis of 40,000,000,000,000 Internet Darkspace Packets
• GraphBLAS on the Edge: Anonymized High Performance Streaming of Network Traffic
• Fast Mapping onto Census Blocks
• Mathematics of Digital Hyperspace
• Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices
• Temporal Correlation of Internet Observatories and Outposts
• Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices