GAMER-2: a GPU-accelerated adaptive mesh refinement code -- accuracy, performance, and scalability
arXiv:1712.07070 · doi:10.1093/mnras/sty2586
Abstract
We present GAMER-2, a GPU-accelerated adaptive mesh refinement (AMR) code for astrophysics. It provides a rich set of features, including adaptive time-stepping, several hydrodynamic schemes, magnetohydrodynamics, self-gravity, particles, star formation, chemistry and radiative processes with GRACKLE, data analysis with yt, and memory pool for efficient object allocation. GAMER-2 is fully bitwise reproducible. For the performance optimization, it adopts hybrid OpenMP/MPI/GPU parallelization and utilizes overlapping CPU computation, GPU computation, and CPU-GPU communication. Load balancing is achieved using a Hilbert space-filling curve on a level-by-level basis without the need to duplicate the entire AMR hierarchy on each MPI process. To provide convincing demonstrations of the accuracy and performance of GAMER-2, we directly compare with Enzo on isolated disk galaxy simulations and with FLASH on galaxy cluster merger simulations. We show that the physical results obtained by different codes are in very good agreement, and GAMER-2 outperforms Enzo and FLASH by nearly one and two orders of magnitude, respectively, on the Blue Waters supercomputers using nodes. More importantly, GAMER-2 exhibits similar or even better parallel scalability compared to the other two codes. We also demonstrate good weak and strong scaling using up to 4096 GPUs and 65,536 CPU cores, and achieve a uniform resolution as high as cells. Furthermore, GAMER-2 can be adopted as an AMR+GPUs framework and has been extensively used for the wave dark matter (DM) simulations. GAMER-2 is open source (available at https://github.com/gamer-project/gamer) and new contributions are welcome.
arXiv comment29 pages, 22 figures. MNRAS accepted. GAMER can be found at https://github.com/gamer-project/gamer
References in corpus (40)
- The Structure of Cold Dark Matter Halos
- The cosmological simulation code GADGET-2
- E pur si muove: Galiliean-invariant cosmological hydrodynamical simulations on a moving mesh
- Cosmological Hydrodynamics with Adaptive Mesh Refinement: a new high resolution code called RAMSES
- A Multi-Code Analysis Toolkit for Astrophysical Simulation Data
- GIZMO: A New Class of Accurate, Mesh-Free Hydrodynamic Simulation Methods
- Athena: A New Code for Astrophysical MHD
- Cosmic Structure as the Quantum Interference of a Coherent Dark Wave
- Enzo: An Adaptive Mesh Refinement Code for Astrophysics
- Adaptive Refinement Tree - a new high-resolution N-body code for cosmological simulations
- Understanding the Core-Halo Relation of Quantum Wave Dark Matter, DM, from 3D Simulations
- A Merger-Driven Scenario for Cosmological Disk Galaxy Formation
- Testing X-ray Measurements of Galaxy Clusters with Cosmological Simulations
- The PLUTO Code for Adaptive Mesh Computations in Astrophysical Fluid Dynamics
- Compressible Turbulence in Galaxy Clusters: Physics and Stochastic Particle Re-acceleration
- Grackle: a Chemistry and Cooling Library for Astrophysics
- CASTRO: A New Compressible Astrophysical Solver. I. Hydrodynamics and Self-Gravity
- Nyx: A Massively Parallel AMR Code for Computational Cosmology
- An Extension of the Athena++ Code Framework for GRMHD Based on Advanced Riemann Solvers and Staggered-Mesh Constrained Transport
- Shaken Snow Globes: Kinematic Tracers of the Multiphase Condensation Cascade in Massive Galaxies, Groups, and Clusters
- A Solution Accurate, Efficient and Stable Unsplit Staggered Mesh Scheme for Three Dimensional Magnetohydrodynamics
- The AGORA High-Resolution Galaxy Simulations Comparison Project. II: Isolated Disk Test
- A Direct Multigrid Poisson Solver for Oct-Tree Adaptive Meshes
- GAMER: a GPU-Accelerated Adaptive Mesh Refinement Code for Astrophysics
- A Parameter Space Exploration of Galaxy Cluster Mergers I: Gas Mixing and the Generation of Cluster Entropy
- Simulating Magnetohydrodynamical Flow with Constrained Transport and Adaptive Mesh Refinement; Algorithms & Tests of the AstroBEAR Code
- Constraining turbulence and conduction in the hot ICM through density perturbations
- Simulating Self-gravitating Hydrodynamic Flows
- Thermal SZ fluctuations in the ICM: probing turbulence and thermodynamics in Coma cluster with
- Mass Transport and Turbulence in Gravitationally Unstable Disk Galaxies II: The Effects of Star Formation Feedback
- Adaptive Mesh Fluid Simulations on GPU
- Recognising Axionic Dark Matter by Compton and de-Broglie Scale Modulation of Pulsar Timing
- Mass Transport and Turbulence in Gravitationally Unstable Disk Galaxies. I: The Case of Pure Self-Gravity
- A Primer on Eulerian Computational Fluid Dynamics for Astrophysics
- Turbulence and cooling in galaxy cluster cores
- On the connection between turbulent motions and particle acceleration in galaxy clusters
- Physical Origins of Gas Motions in Galaxy Cluster Cores: Interpreting Hitomi Observations of the Perseus Cluster
- A New Way to Conserve Total Energy for Eulerian Hydrodynamic Simulations with Self-Gravity
- A GPU accelerated Barnes-Hut Tree Code for FLASH4
- Magnetohydrodynamics with GAMER
Cited by in corpus (37)
- Lyman- forest and non-linear structure characterization in Fuzzy Dark Matter cosmologies
- MPI-AMRVAC 3.0: updates to an open-source simulation framework
- H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing With 3D Adaptive Mesh Refinement and Local Adaptive Time-stepping
- Soliton Random Walk and the Cluster-Stripping Problem in Ultralight Dark Matter
- The Origin of the Stellar Mass Distribution and Multiplicity
- On the Random Motion of Nuclear Objects in a Fuzzy Dark Matter Halo
- Soliton Oscillations and Revised Constraints from Eridanus II of Fuzzy Dark Matter
- Gmunu: Paralleled, grid-adaptive, general-relativistic magnetohydrodynamics in curvilinear geometries in dynamical spacetimes
- How do stars affect DM halos?
- On the Dynamical Heating of Dwarf Galaxies in a Fuzzy Dark Matter Halo
- Quokka: A code for two-moment AMR radiation hydrodynamics on GPUs
- Weak-lensing mass bias in merging galaxy clusters
- Cosmological Simulations of Two-Component Wave Dark Matter
- Gas Density Perturbations in Cool Cores of CLASH Galaxy Clusters
- Turbulent magnetic fields in merging clusters: A case study of Abell 2146
- SCALAR: an AMR code to simulate axion-like dark matter models
- Fuzzy Aquarius: evolution of a Milky-way like system in the Fuzzy Dark Matter scenario
- The Mergers in Abell 2256: Displaced Gas and its Connection to the Radio-emitting Plasma
- cuHARM : a new GPU accelerated GR-MHD code and its application to ADAF disks
- Galactic disc heating by density granulation in fuzzy dark matter simulations
- GPU-Native Adaptive Mesh Refinement with Application to Lattice Boltzmann Simulations
- WENO-Wombat: Scalable Fifth-Order Constrained-Transport Magnetohydrodynamics for Astrophysical Applications
- Large Scale Dynamo in a Primordial Accretion Flow -- An Interpretation from Hydrodynamic Simulation
- Theory and phenomenology of stressed wave-dark-matter soliton
- Self-gravitational Force Calculation of Second Order Accuracy Using Multigrid Method on Nested Grids
- Fuzzy dark matter simulations
- An adaptive mesh, GPU-accelerated, and error minimized special relativistic hydrodynamics code
- Comparison of two different integration methods for the (1+1)-Dimensional Schrödinger-Poisson Equation
- Constraining Merging Galaxy Clusters with X-ray and Lensing Simulations and Observations: The case of Abell 2146
- LightAMR format standard and lossless compression algorithms for adaptive mesh refinement grids: RAMSES use case
- Accelerating wave-propagation algorithms with adaptive mesh refinement using the Graphics Processing Unit (GPU)
- Neural-Network Chemical Emulator for First-Star Formation: Robust Iterative Predictions over a Wide Density Range
- Improved Constraints on Mergers with SZ, Hydrodynamical simulations, Optical, and X-ray (ICM-SHOX). Paper II: Galaxy cluster sample overview
- Emulating the Non-Linear Matter Power-Spectrum in Mixed Axion Dark Matter Models
- libyt: an In Situ Interface Connecting Simulations with yt, Python, and Jupyter Workflows
- OpenGadget3 GPU solver tests
- GPU-native Embedding of Complex Geometries in Adaptive Octree Grids Applied to the Lattice Boltzmann Method