A Community Roadmap for Scientific Workflows Research and Development

Abstract

The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual "Workflows Community Summits" (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

arXiv commentarXiv admin note: substantial text overlap with arXiv:2103.09181

References in corpus (9)

• Packaging research artefacts with RO-Crate
• funcX: A Federated Function Serving Fabric for Science
• Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
• Middleware Building Blocks for Workflow Systems
• SciTokens: Capability-Based Secure Access to Remote Scientific Data
• Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs
• Enabling Machine Learning-Ready HPC Ensembles with Merlin
• Portability of Scientific Workflows in NGS Data Analysis: A Case Study
• Toward Common Components for Open Workflow Systems

Cited by in corpus (7)

• A Terminology for Scientific Workflow Systems
• WfBench: Automated Generation of Scientific Workflow Benchmarks
• Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT
• Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-Practice
• Julia as a unifying end-to-end workflow language on the Frontier exascale system
• Three Practical Workflow Schedulers for Easy Maximum Parallelism
• Pseudonymization at Scale: OLCF's Summit Usage Data Case Study