Simple Data and Workflow Management with the signac Framework

Abstract

Researchers in the field of materials science, chemistry, and computational physics are regularly posed with the challenge of managing large and heterogeneous data spaces. The amount of data increases in lockstep with computational efficiency multiplied by the amount of available computational resources, which shifts the bottleneck in the scientific process from data acquisition to data processing and analysis. We present a framework designed to aid in the integration of various specialized data formats, tools and workflows. The signac framework provides all basic components required to create a well-defined and thus collectively accessible and searchable data space, simplifying data access and modification through a homogeneous data interface that is largely agnostic to the data source, i.e., computation or experiment. The framework's data model is designed to not require absolute commitment to the presented implementation, simplifying adaption into existing data sets and workflows. This approach not only increases the efficiency with which scientific results can be produced, but also significantly lowers barriers for collaborations requiring shared data access.

arXiv comment12 pages, 5 figures

References in corpus (4)

• Strong scaling of general-purpose molecular dynamics simulations on GPUs
• AiiDA: Automated Interactive Infrastructure and Database for Computational Science
• Scalable Metropolis Monte Carlo for simulation of hard shapes
• Massively parallel Monte Carlo for many-particle simulations on GPUs

Cited by in corpus (25)

• AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
• Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows
• Machine Learning Directed Optimization of Classical Molecular Modeling Force Fields
• Efficient implementation of atom-density representations
• Inverse Design of Simple Pair Potentials for the Self-Assembly of Complex Structures
• Crystallization of Binary Nanocrystal Superlattices and the Relevance of Short-Range Attraction
• Programmable patchy particles for materials design
• Pressure-Tunable Photonic Band Gaps in an Entropic Colloidal Crystal
• FCC-to-BCC phase transitions in convex and concave hard particle systems
• Shape and Interaction Decoupling for Colloidal Pre-Assembly
• Programming Interactions in Magnetic Handshake Materials
• Local structural features elucidate crystallization of complex structures
• Cross-stream migration of a Brownian droplet in a polymer solution under Poiseuille flow
• A Python workflow definition for computational materials design
• Change Point Detection of Events in Molecular Simulations using dupin
• mkite: A distributed computing platform for high-throughput materials simulations
• How cooperatively folding are homopolymer molecular knots?
• MC3D: The Materials Cloud computational database of experimentally known stoichiometric inorganics
• Three Practical Workflow Schedulers for Easy Maximum Parallelism
• Exploring the Relationship Between Softness and Excess Entropy in Glass-forming Systems
• Patterns of active dipolar particles in external magnetic fields
• Another source of mesh dependence in topology optimization
• Tuning Stoichiometry to Promote Formation of Binary Colloidal Superlattices
• Topological Design of Heterogeneous Self-Assembly
• Formation of a single quasicrystal upon collision of multiple grains