GRaM-X: A new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

Abstract

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE's Exascale Computing Project (ECP). We use the Z4c formalism to evolve the equations of GR and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D MHD shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D TOV star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of $\sim 40-50\%$ on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF's supercomputer Summit.

22 pages, 8 figures, to be submitted to Classical and Quantum Gravity

References in corpus (15)

• Modules for Experiments in Stellar Astrophysics (MESA)
• The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers
• Magnetorotationally driven Supernovae as the origin of early galaxy $r$-process elements?
• Calibration of Moving Puncture Simulations
• ECHO: an Eulerian Conservative High Order scheme for general relativistic magnetohydrodynamics and magnetodynamics
• WhiskyMHD: a new numerical code for general relativistic magnetohydrodynamics
• Semi-global simulations of the magneto-rotational instability in core collapse supernovae
• General relativistic simulations of compact binary mergers as engines of short gamma-ray bursts
• First 100 ms of a long-lived magnetized neutron star formed in a binary neutron star merger
• Relativistic magnetohydrodynamics in dynamical spacetimes: A new AMR implementation
• Relativistic MHD with Adaptive Mesh Refinement
• Implications of the search for optical counterparts during the first six months of the Advanced LIGO's and Advanced Virgo's third observing run: possible limits on the ejecta mass and binary properties
• A new general relativistic magnetohydrodynamics code for dynamical spacetimes
• The Challenges Ahead for Multimessenger Analyses of Gravitational Waves and Kilonova: a Case Study on GW190425
• Implementation of advanced Riemann solvers in a neutrino-radiation magnetohydrodynamics code in numerical relativity and its application to a binary neutron star merger

Cited by in corpus (6)

• The new discontinuous Galerkin methods based numerical relativity program Nmesh
• Performance-Portable Binary Neutron Star Mergers with AthenaK
• AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes
• Gravitational waves from magnetorotational core-collapse supernovae using 3D GRMHD simulations: effect of rotation and magnetic fields
• Neutron star evolution by combining discontinuous Galerkin and finite volume methods
• A Black Hole is Born: 3D GRMHD Simulation of Black Hole Formation from Core-Collapse