H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing With 3D Adaptive Mesh Refinement and Local Adaptive Time-stepping
arXiv:1912.10192 · doi:10.3847/1538-4365/ac9966
Abstract
General-relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a graphics processing unit (GPU) accelerated GRMHD code \hammer{} with multi-faceted optimizations that, collectively, accelerate computation by 2-5 orders of magnitude for a wide range of applications. Firstly, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the 3 dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Secondly, we demonstrate that local adaptive time-stepping (LAT) on a logarithmic spherical-polar grid accelerates computation by a factor of compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of zone-cycles-per-second-per-node on 5,400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate \hammer{}'s computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of ,,, cells, and a total of billion cells and timesteps, it is among the largest astrophysical simulations ever performed. We find that frame-dragging by the black hole tears up the disk into two independently precessing sub-disks. The innermost sub-disk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields.
arXiv comment21 pages, 10 figures, Accepted by ApJS, for the YouTube playlist see https://youtu.be/rIOjKUfzcvI
References in corpus (94)
- First M87 Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole
- Advection-Dominated Accretion: A Self-Similar Solution
- First Sagittarius A* Event Horizon Telescope Results. I. The Shadow of the Supermassive Black Hole in the Center of the Milky Way
- First M87 Event Horizon Telescope Results. V. Physical Origin of the Asymmetric Ring
- Efficient Generation of Jets from Magnetically Arrested Accretion on a Rapidly Spinning Black Hole
- General Relativistic Magnetohydrodynamic Simulations of Magnetically Choked Accretion Flows around Black Holes
- HARM: A Numerical Scheme for General Relativistic Magnetohydrodynamics
- Advection-Dominated Accretion: Self-Similarity and Bipolar Outflows
- Lense-Thirring Precession and QPOs in Low Mass X-Ray Binaries
- First M87 Event Horizon Telescope Results. VIII. Magnetic Field Structure near The Event Horizon
- General Relativistic Magnetohydrodynamic Simulations of Jet Formation and Large-Scale Propagation from Black Hole Accretion Systems
- A Measurement of the Electromagnetic Luminosity of a Kerr Black Hole
- The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers
- First Sagittarius A* Event Horizon Telescope Results. V. Testing Astrophysical Models of the Galactic Center Black Hole
- An Unsplit Godunov Method for Ideal MHD via Constrained Transport
- Radio-Loudness of Active Galactic Nuclei: Observational Facts and Theoretical Implications
- Low frequency QPO spectra and Lense-Thirring precession
- Global General Relativistic MHD Simulation of a Tilted Black-Hole Accretion Disk
- Formation of Precessing Jets by Tilted Black-hole Discs in 3D General Relativistic MHD Simulations
- First Sagittarius A* Event Horizon Telescope Results. IV. Variability, Morphology, and Black Hole Mass
- Magnetically Driven Accretion Flows in the Kerr Metric I: Models and Overall Structure
- The distribution and cosmic evolution of massive black hole spins
- The Event Horizon General Relativistic Magnetohydrodynamic Code Comparison Project
- Magnetically Driven Jets in the Kerr Metric
- Relativistic Jets Shine through Shocks or Magnetic Reconnection?
- General Relativistic Hydrodynamic Simulation of Accretion Flow from a Stellar Tidal Disruption
- Primitive Variable Solvers for Conservative General Relativistic Magnetohydrodynamics
- Simulations of Magnetized Disks Around Black Holes: Effects of Black Hole Spin, Disk Thickness, and Magnetic Field Geometry
- Aligning spinning black holes and accretion discs
- The Black Hole Accretion Code
- The Influence of Magnetic Field Geometry on the Evolution of Black Hole Accretion Flows: Similar Disks, Drastically Different Jets
- Divergence-Free Adaptive Mesh Refinement for Magnetohydrodynamics
- Black hole flares: ejection of accreted magnetic flux through 3D plasmoid-mediated reconnection
- Total and Jet Blandford-Znajek Power in Presence of Accretion Disk
- Alignment of Magnetized Accretion Disks and Relativistic Jets with Spinning Black Holes
- Optical and Infrared Photometry of the Micro-Quasar GRO J1655-40 in Quiescence
- Direct calculation of the radiative efficiency of an accretion disk around a black hole
- An HLLC Riemann Solver for Relativistic Flows: I. Hydrodynamics
- The non-linear fluid dynamics of a warped accretion disc
- An Extension of the Athena++ Code Framework for GRMHD Based on Advanced Riemann Solvers and Staggered-Mesh Constrained Transport
- Dependence of inner accretion disk stress on parameters: the Schwarzschild case
- A quasi-periodic modulation of the iron line centroid energy in the black hole binary H 1743-322
- Semi-implicit scheme for treating radiation under M1 closure in general relativistic conservative fluid dynamics codes
- On the diffusive propagation of warps in thin accretion discs
- Geometrical constraints on the origin of timing signals from black holes
- Three-Dimensional Simulations of Magnetized Thin Accretion Disks around Black Holes: Stress in the Plunging Region
- Magnetically Driven Accretion Flows in the Kerr Metric II: Structure of the Magnetic Field
- Numerical 3+1 general relativistic magnetohydrodynamics: a local characteristic approach
- Large-Scale Poloidal Magnetic Field Dynamo Leads to Powerful Jets in GRMHD Simulations of Black Hole Accretion with Toroidal Field
- An HLLC Solver for Relativistic Flows -- II. Magnetohydrodynamics
- Fornax: a Flexible Code for Multiphysics Astrophysical Simulations
- A unified accretion-ejection paradigm for Black Hole X-ray Binaries
- On the Bardeen-Petterson Effect in black hole accretion discs
- Tearing up the disc: how black holes accrete
- WHAM: A WENO-based general relativistic numerical scheme I: Hydrodynamics
- High-order conservative reconstruction schemes for finite volume methods in cylindrical and spherical coordinates
- Bardeen-Petterson Alignment, Jets and Magnetic Truncation in GRMHD Simulations of Tilted Thin Accretion Discs
- Hydrodynamic Simulations of Tilted Thick-Disk Accretion onto a Kerr Black Hole
- The Disc-Jet Symbiosis Emerges: Modeling the Emission of Sagittarius A* with Electron Thermodynamics
- Accelerating AGN jets to parsec scales using general relativistic MHD simulations
- Radiative, two-temperature simulations of low luminosity black hole accretion flows in general relativity
- A five-wave HLL Riemann solver for relativistic MHD
- The evolution of a warped disc around a Kerr black hole
- A rapidly-changing jet orientation in the stellar-mass black hole V404 Cygni
- Disc Tearing and Bardeen-Petterson Alignment in GRMHD Simulations of Highly Tilted Thin Accretion Discs
- Formation of Magnetically Truncated Accretion Disks in 3D Radiation-Transport Two-Temperature GRMHD Simulations
- Modeling non-thermal emission from the jet-launching region of M 87 with adaptive mesh refinement
- General Relativistic Magnetohydrodynamic Simulations of the Hard State as a Magnetically-Dominated Accretion Flow
- Observational signatures of disk and jet misalignment in images of accreting black holes
- GAMER-2: a GPU-accelerated adaptive mesh refinement code -- accuracy, performance, and scalability
- Two-Temperature GRRMHD Simulations of M87
- GRMHD simulations of accretion onto Sgr A*: How important are radiative losses?
- General relativistic resistive magnetohydrodynamics with robust primitive variable recovery for accretion disk simulations
- A mechanism for hysteresis in black hole binary state transitions
- GRRMHD Simulations of Tidal Disruption Event Accretion Disks around Supermassive Black Holes: Jet Formation, Spectra, and Detectability
- Black hole to breakout: 3D GRMHD simulations of collapsar jets reveal a wide range of transients
- Where is the Radiation Edge in Magnetized Black Hole Accretion discs?
- Monte-Carlo closure for moment-based transport schemes in general relativistic radiation hydrodynamics simulations
- Detection of the first infra-red quasi periodic oscillation in a black hole X-ray binary
- Tidal Disruption Disks Formed and Fed by Stream-Stream and Stream-Disk Interactions in Global GRHD Simulations
- The Intrinsic Eddington Ratio Distribution of Active Galactic Nuclei in Star-forming Galaxies from the Sloan Digital Sky Survey
- Evolving Nonthermal Electrons in Simulations of Black Hole Accretion
- A unified accretion-ejection paradigm for black hole X-ray binaries. III. Spectral signatures of hybrid disk configurations
- Relativistic resistive magnetohydrodynamic reconnection and plasmoid formation in merging flux tubes
- A unified accretion-ejection paradigm for black hole X-ray binaries. II. Observational signatures of jet-emitting disks
- Resolution Dependence of Magnetorotational Turbulence in the Isothermal Stratified Shearing Box
- Is the Bardeen-Petterson effect responsible for the warping and precession in NGC 4258?
- Self-consistent spectra from radiative GRMHD simulations of accretion onto Sgr A*
- Spectral and Imaging properties of Sgr A* from High-Resolution 3D GRMHD Simulations with Radiative Cooling
- On the jet-ejecta interaction in 3D GRMHD simulations of binary neutron star merger aftermath
- Bardeen-Petterson effect and the disk structure of the Seyfert galaxy NGC 1068
- Numerical relativity in spherical coordinates: A new dynamical spacetime and general relativistic MHD evolution framework for the Einstein Toolkit
- A Constrained Transport Method for the Solution of the Resistive Relativistic MHD Equations
- Numerical relativity in spherical coordinates with the Einstein Toolkit
Cited by in corpus (45)
- Large-Scale Poloidal Magnetic Field Dynamo Leads to Powerful Jets in GRMHD Simulations of Black Hole Accretion with Toroidal Field
- Key Science Goals for the Next-Generation Event Horizon Telescope
- Disc Tearing and Bardeen-Petterson Alignment in GRMHD Simulations of Highly Tilted Thin Accretion Discs
- Observational signatures of disk and jet misalignment in images of accreting black holes
- Black hole to breakout: 3D GRMHD simulations of collapsar jets reveal a wide range of transients
- General relativistic MHD simulations of non-thermal flaring in Sagittarius A*
- Magnetically Modified Spherical Accretion in GRMHD: Reconnection-Driven Convection and Jet Propagation
- Spectral and Imaging properties of Sgr A* from High-Resolution 3D GRMHD Simulations with Radiative Cooling
- Rapid Black Hole Spin-down by Thick Magnetically Arrested Disks
- Jet Formation in 3D GRMHD Simulations of Bondi-Hoyle-Lyttleton Accretion
- Radiation Transport Two-Temperature GRMHD Simulations of Warped Accretion Disks
- Nozzle Shocks, Disk Tearing and Streamers Drive Rapid Accretion in 3D GRMHD Simulations of Warped Thin Disks
- Gmunu: Paralleled, grid-adaptive, general-relativistic magnetohydrodynamics in curvilinear geometries in dynamical spacetimes
- Collapsar Black Holes are Likely Born Slowly Spinning
- Strongly magnetized accretion with low angular momentum produces a weak jet
- Accretion Flow Morphology in Numerical Simulations of Black Holes from the ngEHT Model Library: The Impact of Radiation Physics
- An extension of Gmunu: General-relativistic resistive magnetohydrodynamics based on staggered-meshed constrained transport with elliptic cleaning
- H-AMR FORGE'd in FIRE I: Magnetic state transitions, jet launching and radiative emission in super-Eddington, highly magnetized quasar disks formed from cosmological initial conditions
- Winds and Disk Turbulence Exert Equal Torques on Thick Magnetically Arrested Disks
- Magnetically-Driven Neutron-Rich Ejecta Unleashed: Global 3D Neutrino-General Relativistic Magnetohydrodynamic Simulations of Collapsars Probe the Conditions for r-process Nucleosynthesis
- A Phase Lag between Disk and Corona in GRMHD Simulations of Precessing Tilted Accretion Disks
- The stringent upper limit on jet power in the persistent soft state source 4U~1957+11
- Toward General-Relativistic Magnetohydrodynamics Simulations in Stationary Non-Vacuum Spacetimes
- State-of-the-Art Collapsar Jet Simulations Imply Undetectable Subphotospheric Neutrinos
- Misaligned magnetized accretion flows onto spinning black holes: Magneto-spin alignment, outflow power, and intermittent jets
- Jetted and Turbulent Stellar Deaths: New LVK-Detectable Gravitational Wave Sources
- Subphotospheric emission from short gamma-ray bursts: Protons mold the multi-messenger signals
- The plunging region of a thin accretion disc around a Schwarzschild black hole
- Evidence for Low Universal Equilibrium Black Hole Spin in Luminous Magnetically Arrested Disks
- General relativistic force-free electrodynamics with a discontinuous Galerkin-finite difference hybrid method
- On the Universality of Energy Extraction from Black Hole Spacetimes
- Extreme scale height variations and nozzle shocks in warped disks
- Tilted Accretion Disks
- GRMHD simulations of black hole accretion variabilities: Implications to hard state X-ray binary transients
- Amoeba: An AGN Model of Optical Emissions Beyond steady-state Accretion discs
- General relativistic hydrodynamics code for dynamical spacetimes with curvilinear coordinates, tabulated equations of state, and neutrino physics
- Diverse Emission Patterns from Precessing Super-Eddington Disks Formed in Tidal Disruption Events
- DX100: A Programmable Data Access Accelerator for Indirection
- Fast Transients from Magnetic Disks Around Non-Spinning Collapsar Black Holes
- Black Hole Spin-down in Collapsars in 3D Neutrino Transport GRMHD Simulations
- Extreme magnetic fields around black holes
- CAFE-AMR: A computational MHD Solar Physics simulation tool that uses AMR
- Relativistic Magnetohydrodynamic Simulations of Giant Magnetar Bursts
- GRMHD Simulations of Magnetized Accretion Disk/Jet: Variabilities of Black Holes and Spectral Energy Distributions in Magnetic States
- Constraints on the accretion properties of quasi-periodic erupters from GRMHD simulations