StreamFlow: cross-breeding cloud with HPC

Abstract

Workflows are among the most commonly used tools in a variety of execution environments. Many of them target a specific environment; few of them make it possible to execute an entire workflow in different environments, e.g. Kubernetes and batch clusters. We present a novel approach to workflow execution, called StreamFlow, that complements the workflow graph with the declarative description of potentially complex execution environments, and that makes it possible the execution onto multiple sites not sharing a common data space. StreamFlow is then exemplified on a novel bioinformatics pipeline for single-cell transcriptomic data analysis workflow.

30 pages - 2020 IEEE Transactions on Emerging Topics in Computing

References in corpus (1)

• OCCAM: a flexible, multi-purpose and extendable HPC cluster

Cited by in corpus (8)

• Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
• Containerisation for High Performance Computing Systems: Survey and Prospects
• Recording provenance of workflow runs with RO-Crate
• A Terminology for Scientific Workflow Systems
• EuroHPC SPACE CoE: Redesigning Scalable Parallel Astrophysical Codes for Exascale
• Introducing SWIRL: An Intermediate Representation Language for Scientific Workflows
• Dynamic Solutions for Hybrid Quantum-HPC Resource Allocation
• Three ways to share a QPU: Scheduling strategies for hybrid Quantum-HPC applications