Exploratory Data Science on Supercomputers for Quantum Mechanical Calculations

Abstract

Literate programming - the bringing together of program code and natural language narratives - has become a ubiquitous approach in the realm of data science. This methodology is appealing as well for the domain of Density Functional Theory (DFT) calculations, particularly for interactively developing new methodologies and workflows. However, effective use of literate programming is hampered by old programming paradigms and the difficulties associated with using High Performance Computing (HPC) resources. Here we present two Python libraries that aim to remove these hurdles. First, we describe the PyBigDFT library, which can be used to setup materials or molecular systems and provides high-level access to the wavelet based BigDFT code. We then present the related remotemanager library, which is able to serialize and execute arbitrary Python functions on remote supercomputers. We show how together these libraries enable transparent access to HPC based DFT calculations and can serve as building blocks for rapid prototyping and data exploration.

References in corpus (13)

• AFLOW: An automatic framework for high-throughput materials discovery
• Recent developments in the PySCF program package
• fastai: A Layered API for Deep Learning
• NWChem: Past, Present, and Future
• AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
• Parsl: Pervasive Parallel Programming in Python
• Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows
• AiiDAlab -- an ecosystem for developing, executing, and sharing scientific workflows
• Surface science using coupled cluster theory via local Wannier functions and in-RPA-embedding: the case of water on graphitic carbon nitride
• Eliciting Best Practices for Collaboration with Computational Notebooks
• TREXIO: A File Format and Library for Quantum Chemistry
• The MRChem multiresolution analysis code for molecular electronic structure calculations: performance and scaling properties
• Fragment Approach to Constrained Density Functional Theory Calculations using Daubechies Wavelets