Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows

Abstract

Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look at the workflow engine powering AiiDA, a widely adopted, highly flexible and database-backed informatics infrastructure with an emphasis on data reproducibility. We detail many of the design choices that were made which were informed by several important goals: the ability to scale from running on individual laptops up to high-performance supercomputers, managing jobs with runtimes spanning from fractions of a second to weeks and scaling up to thousands of jobs concurrently, and all this while maximising robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput computational science. As well as the architecture, we outline important API design choices made to give workflow writers a great deal of liberty whilst guiding them towards writing robust and modular workflows, ultimately enabling them to encode their scientific knowledge to the benefit of the wider scientific community.

References in corpus (6)

• AFLOW: An automatic framework for high-throughput materials discovery
• Materials Cloud, a platform for open computational science
• AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
• Simple Data and Workflow Management with the signac Framework
• The MAterials Simulation Toolkit (MAST) for atomistic modeling of defects and diffusion
• kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows

Cited by in corpus (42)

• AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
• A Deep Dive into Machine Learning Density Functional Theory for Materials Science and Chemistry
• OPTIMADE, an API for exchanging materials data
• How to verify the precision of density-functional-theory implementations via reproducible and universal workflows
• Thermodynamics and dielectric response of ${\text{BaTiO}}_3$ by data-driven modeling
• El Agente: An Autonomous Agent for Quantum Chemistry
• Developments and applications of the OPTIMADE API for materials discovery, design, and data exchange
• Machine-learning structural reconstructions for accelerated point defect calculations
• Universal materials model of deep-learning density functional theory Hamiltonian
• Common workflows for computing material properties using different quantum engines
• Fermi energy determination for advanced smearing techniques
• Orbital-resolved DFT+U for molecules and solids
• Abinit 2025: New Capabilities for the Predictive Modeling of Solids and Nanomaterials
• The Wannier Function Software Ecosystem for Materials Simulations
• Automated all-functionals infrared and Raman spectra
• Projectability disentanglement for accurate and automated electronic-structure Hamiltonians
• Density functional Bogoliubov-de Gennes analysis of superconducting Nb and Nb(110) surfaces
• Understanding the role of Hubbard corrections in the rhombohedral phase of BaTiO${}_3$
• Accelerating Cathode Material Discovery through Ab Initio Random Structure Searching
• Microscopic picture of paraelectric perovskites from structural prototypes
• Towards high-throughput many-body perturbation theory: efficient algorithms and automated workflows
• Automated mixing of maximally localized Wannier functions into target manifolds
• Exploring the Cs-Te phase space via high-throughput density-functional theory calculations beyond the generalized-gradient approximation
• Charting the landscape of Bardeen-Cooper-Schrieffer superconductors in experimentally known compounds
• Training-free hyperparameter optimization of neural networks for electronic structures in matter
• First-principles Hubbard parameters with automated and reproducible workflows
• A Fully Quantum-Mechanical Treatment for Kaolinite
• Interoperable Workflows by Exchanging Grid-Based Data between Quantum-Chemical Program Packages
• A Python workflow definition for computational materials design
• Automated computational workflows for muon spin spectroscopy
• mkite: A distributed computing platform for high-throughput materials simulations
• The AiiDA-Spirit plugin for automated spin-dynamics simulations and multi-scale modelling based on first-principles calculations
• MC3D: The Materials Cloud computational database of experimentally known stoichiometric inorganics
• Robust Wannierization including magnetization and spin-orbit coupling via projectability disentanglement
• Spectral Operator Representations
• Strategies to search for two-dimensional materials with long spin qubit coherence time
• Exploratory Data Science on Supercomputers for Quantum Mechanical Calculations
• Ab initio functional-independent calculations of the clamped Pockels tensor of tetragonal barium titanate
• AMaRaNTA: Automated First-Principles Exchange Parameters In 2D Magnets
• Making atomistic materials calculations accessible with the AiiDAlab Quantum ESPRESSO app
• Uncoupling System and Environment Simulation Cells for Fast-Scaling Modeling of Complex Continuum Embeddings
• Score-based diffusion models for accurate crystal-structure inpainting and reconstruction of hydrogen positions