Parsl: Pervasive Parallel Programming in Python

Abstract

High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs. Here, we present Parsl, a parallel scripting library that augments Python with simple, scalable, and flexible constructs for encoding parallelism. These constructs allow Parsl to construct a dynamic dependency graph of components that it can then execute efficiently on one or many processors. Parsl is designed for scalability, with an extensible set of executors tailored to different use cases, such as low-latency, high-throughput, or extreme-scale execution. We show, via experiments on the Blue Waters supercomputer, that Parsl executors can allow Python scripts to execute components with as little as 5 ms of overhead, scale to more than 250 000 workers across more than 8000 nodes, and process upward of 1200 tasks per second. Other Parsl features simplify the construction and execution of composite programs by supporting elastic provisioning and scaling of infrastructure, fault-tolerant execution, and integrated wide-area data management. We show that these capabilities satisfy the needs of many-task, interactive, online, and machine learning applications in fields such as biology, cosmology, and materials science.

References in corpus (1)

• Ray: A Distributed Framework for Emerging AI Applications

Cited by in corpus (34)

• AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance
• funcX: A Federated Function Serving Fabric for Science
• Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language
• Applying the FAIR Principles to computational workflows
• A survey of biodiversity informatics: Concepts, practices, and challenges
• Colmena: Scalable Machine-Learning-Based Steering of Ensemble Simulations for High Performance Computing
• funcX: Federated Function as a Service for Science
• libEnsemble: A Library to Coordinate the Concurrent Evaluation of Dynamic Ensembles of Calculations
• A Unified Catalog-level Reanalysis of Stage-III Cosmic Shear Surveys
• A Terminology for Scientific Workflow Systems
• Reproducible and Portable Big Data Analytics in the Cloud
• Accelerating Electronic Stopping Power Predictions by 10 Million Times with a Combination of Time-Dependent Density Functional Theory and Machine Learning
• PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs
• Cloud Services Enable Efficient AI-Guided Simulation Workflows across Heterogeneous Resources
• Auriga Streams I: disrupting satellites surrounding Milky Way-mass haloes at multiple resolutions
• Supercomputing with MPI meets the Common Workflow Language standards: an experience report
• Design and Evaluation of a Simple Data Interface for Efficient Data Transfer Across Diverse Storage
• RosneT: A Block Tensor Algebra Library for Out-of-Core Quantum Computing Simulation
• RAPTOR: Ravenous Throughput Computing
• UniFaaS: Programming across Distributed Cyberinfrastructure with Federated Function Serving
• Toward a Framework for Integrative, FAIR, and Reproducible Management of Data on the Dynamic Balance of Microbial Communities
• Distributed statistical inference with pyhf enabled through funcX
• Portable, heterogeneous ensemble workflows at scale using libEnsemble
• Pandemic Drugs at Pandemic Speed: Infrastructure for Accelerating COVID-19 Drug Discovery with Hybrid Machine Learning- and Physics-based Simulations on High Performance Computers
• Analysis of Workflow Schedulers in Simulated Distributed Environments
• The BioExcel methodology for developing dynamic, scalable, reliable and portable computational biomolecular workflows
• Exploratory Data Science on Supercomputers for Quantum Mechanical Calculations
• Towards Enabling I/O Awareness in Task-based Programming Models
• Scaling on Frontier: Uncertainty Quantification Workflow Applications using ExaWorks to Enable Full System Utilization
• The Common Workflow Scheduler Interface: Status Quo and Future Plans
• High Throughput Training of Deep Surrogates from Large Ensemble Runs
• StreamGuard: Low-Overhead Resilience for Real-time HPC Data Streams
• Extreme Scale Survey Simulation with Python Workflows
• Extended Abstract: Productive Parallel Programming with Parsl