Overcoming Orchestration Bottlenecks at Exascale: A Decentralized, Policy-Driven Approach for Sim-AI Ensembles
arXiv:2607.12211
The paper presents EnsembleLauncher, a decentralized, hierarchical workflow orchestrator that scales to exascale systems and allows programmable scheduling policies to improve resource utilization for large, heterogeneous simulation‑AI ensembles.
Abstract
Scientific computing is increasingly shifting from monolithic applications to coupled simulation-AI workflows composed of highly heterogeneous tasks with diverse hardware, scale, and runtime requirements. As these workflows scale to leadership-class systems, the resulting extreme ensemble sizes and task variability can create orchestration bottlenecks. System-level schedulers are often configured for limited throughput, while workflow tools face scalability issues due to rigid control-plane topologies and static scheduling heuristics. We introduce EnsembleLauncher, a recursively hierarchical workflow orchestrator for exascale systems, featuring a fully decentralized control plane and a programmable scheduling policy interface. On the Aurora supercomputer, EnsembleLauncher successfully scales to the entire machine with up to eight million serial tasks, outperforming state-of-the-art tools by more than four times. Additionally, we implement a programmable scheduling interface and demonstrate a significant impact of scheduling policies on resource utilization for high-variance ensembles and active learning pipelines representative of modern coupled simulation-AI workflows.