Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices

Abstract

Hypersparse matrices are a powerful enabler for a variety of network, health, finance, and social applications. Hierarchical hypersparse GraphBLAS matrices enable rapid streaming updates while preserving algebraic analytic power and convenience. In many contexts, the rate of these updates sets the bounds on performance. This paper explores hierarchical hypersparse update performance on a variety of hardware with identical software configurations. The high-level language bindings of the GraphBLAS readily enable performance experiments on simultaneous diverse hardware. The best single process performance measured was 4,000,000 updates per second. The best single node performance measured was 170,000,000 updates per second. The hardware used spans nearly a decade and allows a direct comparison of hardware improvements for this computation over this time range; showing a 2x increase in single-core performance, a 3x increase in single process performance, and a 5x increase in single node performance. Running on nearly 2,000 MIT SuperCloud nodes simultaneously achieved a sustained update rate of over 200,000,000,000 updates per second. Hierarchical hypersparse GraphBLAS allows the MIT SuperCloud to analyze extremely large streaming network data sets.

arXiv comment6 pages, 5 figures, 32 references, accepted to IEEE HPEC 2021. arXiv admin note: text overlap with arXiv:2001.06935

References in corpus (12)

• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• Parallel Sparse Matrix-Matrix Multiplication and Indexing: Implementation and Experiments
• Static Graph Challenge: Subgraph Isomorphism
• Streaming Graph Challenge: Stochastic Block Partition
• Achieving 100,000,000 database inserts per second using Accumulo and D4M
• Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M
• 75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices
• Multi-Temporal Analysis and Scaling Relations of 100,000,000,000 Network Packets
• Zero Botnets: An Observe-Pursue-Counter Approach
• Benchmarking SciDB Data Import on HPC Systems
• Large Scale Parallelization Using File-Based Communications
• Optimizing Xeon Phi for Interactive Data Analysis

Cited by in corpus (5)

• Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays
• Temporal Correlation of Internet Observatories and Outposts
• Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices
• What is Normal? A Big Data Observational Science Model of Anonymized Internet Traffic
• Teaching Network Traffic Matrices in an Interactive Game Environment