Python Implementation of the Dynamic Distributed Dimensional Data Model

Abstract

Python has become a standard scientific computing language with fast-growing support of machine learning and data analysis modules, as well as an increasing usage of big data. The Dynamic Distributed Dimensional Data Model (D4M) offers a highly composable, unified data model with strong performance built to handle big data fast and efficiently. In this work we present an implementation of D4M in Python. $D4M.py$ implements all foundational functionality of D4M and includes Accumulo and SQL database support via Graphulo. We describe the mathematical background and motivation, an explanation of the approaches made for its fundamental functions and building blocks, and performance results which compare $D4M.py$'s performance to D4M-MATLAB and D4M.jl.

arXiv comment8 pages, 7 figures, accepted to HPEC 2022

References in corpus (10)

• Scikit-learn: Machine Learning in Python
• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Array Programming with NumPy
• TensorFlow: A system for large-scale machine learning
• Graphulo: Linear Algebra Graph Kernels for NoSQL Databases
• Achieving 100,000,000 database inserts per second using Accumulo and D4M
• D4M: Bringing Associative Arrays to Database Engines
• Julia Implementation of the Dynamic Distributed Dimensional Data Model
• Benchmarking SciDB Data Import on HPC Systems
• Rapid Sequence Identification of Potential Pathogens Using Techniques from Sparse Linear Algebra

Cited by in corpus (1)

• Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays