SWIFT: Maintaining weak-scalability with a dynamic range of $10^4$ in time-step size to harness extreme adaptivity

Abstract

Cosmological simulations require the use of a multiple time-stepping scheme. Without such a scheme, cosmological simulations would be impossible due to their high level of dynamic range; over eleven orders of magnitude in density. Such a large dynamic range leads to a range of over four orders of magnitude in time-step, which presents a significant load-balancing challenge. In this work, the extreme adaptivity that cosmological simulations present is tackled in three main ways through the use of the code SWIFT. First, an adaptive mesh is used to ensure that only the relevant particles are interacted in a given time-step. Second, task-based parallelism is used to ensure efficient load-balancing within a single node, using pthreads and SIMD vectorisation. Finally, a domain decomposition strategy is presented, using the graph domain decomposition library METIS, that bisects the work that must be performed by the simulation between nodes using MPI. These three strategies are shown to give SWIFT near-perfect weak-scaling characteristics, only losing 25% performance when scaling from 1 to 4096 cores on a representative problem, whilst being more than 30x faster than the de-facto standard Gadget-2 code.

References in corpus (13)

• The EAGLE project: Simulating the evolution and assembly of galaxies and their environments
• Simulating Galaxy Formation with the IllustrisTNG Model
• The EAGLE simulations of galaxy formation: calibration of subgrid physics and model variations
• GIZMO: A New Class of Accurate, Mesh-Free Hydrodynamic Simulation Methods
• FIRE-2 Simulations: Physics versus Numerics in Galaxy Formation
• Smoothed Particle Hydrodynamics and Magnetohydrodynamics
• Grackle: a Chemistry and Cooling Library for Astrophysics
• Gasoline2: A Modern SPH Code
• The EAGLE simulations of galaxy formation: the importance of the hydrodynamics scheme
• The Dynamical and Chemical Evolution of Dwarf Spheroidal Galaxies
• SEREN - A new SPH code for star and planet formation simulations
• SWIFT: Using task-based parallelism, fully asynchronous communication, and graph partition-based domain decomposition for strong scaling on more than 100,000 cores
• SWIFT: Fast algorithms for multi-resolution SPH on multi-core architectures

Cited by in corpus (3)

• The importance of black hole repositioning for galaxy formation simulations
• Modelling of ionising feedback with Smoothed Particle Hydrodynamics and Monte Carlo Radiative Transfer on a Voronoi grid
• A Hybrid MPI+Threads Approach to Particle Group Finding Using Union-Find