Achieving 100,000,000 database inserts per second using Accumulo and D4M

Abstract

The Apache Accumulo database is an open source relaxed consistency database that is widely used for government applications. Accumulo is designed to deliver high performance on unstructured data such as graphs of network data. This paper tests the performance of Accumulo using data from the Graph500 benchmark. The Dynamic Distributed Dimensional Data Model (D4M) software is used to implement the benchmark on a 216-node cluster running the MIT SuperCloud software stack. A peak performance of over 100,000,000 database inserts per second was achieved which is 100x larger than the highest previously published value for any other database. The performance scales linearly with the number of ingest clients, number of database servers, and data size. The performance was achieved by adapting several supercomputing techniques to this application: distributed arrays, domain decomposition, adaptive load balancing, and single-program-multiple-data programming.

6 pages; to appear in IEEE High Performance Extreme Computing (HPEC) 2014

References in corpus (1)

• D4M 2.0 Schema: A General Purpose High Performance Schema for the Accumulo Database

Cited by in corpus (19)

• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• Enabling On-Demand Database Computing with MIT SuperCloud Database Management System
• PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms
• Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M
• 75,000,000,000 Streaming Inserts/Second Using Hierarchical Hypersparse GraphBLAS Matrices
• Benchmarking Data Analysis and Machine Learning Applications on the Intel KNL Many-Core Processor
• Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs
• Loginson: a transform and load system for very large scale log analysis in large IT infrastructures
• Lustre, Hadoop, Accumulo
• Best of Both Worlds: High Performance Interactive and Batch Launching
• Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers
• Mathematics of Digital Hyperspace
• KISS methodologies for network management and anomaly detection
• Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices
• Database Operations in D4M.jl
• Securing HPC using Federated Authentication
• Python Implementation of the Dynamic Distributed Dimensional Data Model
• HPC with Enhanced User Separation
• DBOS Network Sensing: A Web Services Approach to Collaborative Awareness