Parthenon -- a performance portable block-structured adaptive mesh refinement framework

Abstract

On the path to exascale the landscape of computer device architectures and corresponding programming models has become much more diverse. While various low-level performance portable programming models are available, support at the application level lacks behind. To address this issue, we present the performance portable block-structured adaptive mesh refinement (AMR) framework Parthenon, derived from the well-tested and widely used Athena++ astrophysical magnetohydrodynamics code, but generalized to serve as the foundation for a variety of downstream multi-physics codes. Parthenon adopts the Kokkos programming model, and provides various levels of abstractions from multi-dimensional variables, to packages defining and separating components, to launching of parallel compute kernels. Parthenon allocates all data in device memory to reduce data movement, supports the logical packing of variables and mesh blocks to reduce kernel launch overhead, and employs one-sided, asynchronous MPI calls to reduce communication overhead in multi-node simulations. Using a hydrodynamics miniapp, we demonstrate weak and strong scaling on various architectures including AMD and NVIDIA GPUs, Intel and AMD x86 CPUs, IBM Power9 CPUs, as well as Fujitsu A64FX CPUs. At the largest scale on Frontier (the first TOP500 exascale machine), the miniapp reaches a total of $1.7\times 10^{13}$ zone-cycles/s on 9,216 nodes (73,728 logical GPUs) at ~92% weak scaling parallel efficiency (starting from a single node). In combination with being an open, collaborative project, this makes Parthenon an ideal framework to target exascale simulations in which the downstream developers can focus on their specific application rather than on the complexity of handling massively-parallel, device-accelerated AMR.

17 pages, 11 figures, accepted for publication in IJHPCA, Codes available at https://github.com/parthenon-hpc-lab

References in corpus (11)

• The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers
• The RAGE radiation-hydrodynamic code
• Full Transport Model of GW170817-Like Disk Produces a Blue Kilonova
• A Survey of High Level Frameworks in Block-Structured Adaptive Mesh Refinement Packages
• The sonic scale revealed by the world's largest supersonic turbulence simulation
• AMReX: Block-Structured Adaptive Mesh Refinement for Multiphysics Applications
• Ab Initio Horizon-Scale Simulations of Magnetically Arrested Accretion in Sagittarius A* Fed by Stellar Winds
• AGN jet feedback on a moving mesh: gentle cluster heating by weak shocks and lobe disruption
• $\nu \text{bhlight}$: Radiation GRMHD for Neutrino-Driven Accretion Flows
• Enabling particle applications for exascale computing platforms
• As a matter of tension -- kinetic energy spectra in MHD turbulence

Cited by in corpus (10)

• The Launching of Cold Clouds by Galaxy Outflows V: The Role of Anisotropic Thermal Conduction
• Performance-Portable Binary Neutron Star Mergers with AthenaK
• The properties of magnetised cold filaments in a cool-core galaxy cluster
• AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes
• KHARMA: Flexible, Portable Performance for GRMHD
• XMAGNET: Velocity structure functions of active galactic nucleus-driven turbulence in the multiphase intracluster medium
• Stencil Computations on AMD and Nvidia Graphics Processors: Performance and Tuning Strategies
• Neutron star evolution by combining discontinuous Galerkin and finite volume methods
• Benchmarking with Supernovae: A Performance Study of the FLASH Code
• An Improved Fit to the Density Distribution in Supersonic Isothermal Turbulence