D4M 2.0 Schema: A General Purpose High Performance Schema for the Accumulo Database

Abstract

Non-traditional, relaxed consistency, triple store databases are the backbone of many web companies (e.g., Google Big Table, Amazon Dynamo, and Facebook Cassandra). The Apache Accumulo database is a high performance open source relaxed consistency database that is widely used for government applications. Obtaining the full benefits of Accumulo requires using novel schemas. The Dynamic Distributed Dimensional Data Model (D4M)[http://d4m.mit.edu] provides a uniform mathematical framework based on associative arrays that encompasses both traditional (i.e., SQL) and non-traditional databases. For non-traditional databases D4M naturally leads to a general purpose schema that can be used to fully index and rapidly query every unique string in a dataset. The D4M 2.0 Schema has been applied with little or no customization to cyber, bioinformatics, scientific citation, free text, and social media data. The D4M 2.0 Schema is simple, requires minimal parsing, and achieves the highest published Accumulo ingest rates. The benefits of the D4M 2.0 Schema are independent of the D4M interface. Any interface to Accumulo can achieve these benefits by using the D4M 2.0 Schema

6 pages; IEEE HPEC 2013

References in corpus (1)

• MatlabMPI

Cited by in corpus (29)

• Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
• Computing on Masked Data: a High Performance Method for Improving Big Data Veracity
• Graphulo: Linear Algebra Graph Kernels for NoSQL Databases
• Graphulo Implementation of Server-Side Sparse Matrix Multiply in the Accumulo Database
• Achieving 100,000,000 database inserts per second using Accumulo and D4M
• D4M: Bringing Associative Arrays to Database Engines
• Measuring the Impact of Spectre and Meltdown
• Associative Array Model of SQL, NoSQL, and NewSQL Databases
• Big Data Dimensional Analysis
• Polystore Mathematics of Relational Algebra
• Using a Power Law Distribution to describe Big Data
• Big Data Strategies for Data Center Infrastructure Management Using a 3D Gaming Platform
• LLMapReduce: Multi-Level Map-Reduce for High Performance Data Analysis
• Large Scale Organization and Inference of an Imagery Dataset for Public Safety
• From NoSQL Accumulo to NewSQL Graphulo: Design and Utility of Graph Algorithms inside a BigTable Database
• Computing on Masked Data to improve the Security of Big Data
• Julia Implementation of the Dynamic Distributed Dimensional Data Model
• Genetic Sequence Matching Using D4M Big Data Approaches
• Benchmarking Data Analysis and Machine Learning Applications on the Intel KNL Many-Core Processor
• Benchmarking SciDB Data Import on HPC Systems
• Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs
• Mathematics of Digital Hyperspace
• Lessons Learned from a Decade of Providing Interactive, On-Demand High Performance Computing to Scientists and Engineers
• Best of Both Worlds: High Performance Interactive and Batch Launching
• Database Operations in D4M.jl
• D4M 3.0: Extended Database and Language Capabilities
• Securing HPC using Federated Authentication
• HPC with Enhanced User Separation
• Linear Systems over Join-Blank Algebras