Scaling Lattice QCD beyond 100 GPUs

Abstract

Over the past five years, graphics processing units (GPUs) have had a transformational effect on numerical lattice quantum chromodynamics (LQCD) calculations in nuclear and particle physics. While GPUs have been applied with great success to the post-Monte Carlo "analysis" phase which accounts for a substantial fraction of the workload in a typical LQCD calculation, the initial Monte Carlo "gauge field generation" phase requires capability-level supercomputing, corresponding to O(100) GPUs or more. Such strong scaling has not been previously achieved. In this contribution, we demonstrate that using a multi-dimensional parallelization strategy and a domain-decomposed preconditioner allows us to scale into this regime. We present results for two popular discretizations of the Dirac operator, Wilson-clover and improved staggered, employing up to 256 GPUs on the Edge cluster at Lawrence Livermore National Laboratory.

11 pages, 10 figures, to appear in the proceedings of the 2011 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC'11)

References in corpus (6)

• First results from 2+1 dynamical quark flavors on an anisotropic lattice: light-hadron spectroscopy and setting the strange-quark mass
• Lattice QCD as a video game
• Multi-mass solvers for lattice QCD on GPUs
• Domain Decomposition method on GPU cluster
• Efficient implementation of the overlap operator on multi-GPUs
• GPU-Based Conjugate Gradient Solver for Lattice QCD with Domain-Wall Fermions

Cited by in corpus (67)

• A percent-level determination of the nucleon axial coupling from Quantum Chromodynamics
• Valence parton distribution of pion from lattice QCD: Approaching continuum
• Up, down, and strange nucleon axial form factors from lattice QCD
• Gluon gravitational structure of hadrons of different spin
• Towards grounding nuclear physics in QCD
• Heavy physics contributions to neutrinoless double beta decay from QCD
• High-precision calculation of the strange nucleon electromagnetic form factors
• On the Feynman-Hellmann Theorem in Quantum Field Theory and the Calculation of Matrix Elements
• Two-nucleon S-wave interactions at the $SU(3)$ flavor-symmetric point with $m_{ud}\simeq m_s^{\mathrm{phys}}$: a first lattice QCD calculation with the stochastic Laplacian Heaviside method
• Scattering of two and three physical pions at maximal isospin from lattice QCD
• Sigma terms and strangeness content of the nucleon with $N_f=2+1+1$ twisted mass fermions
• Hadron-Hadron Interactions from $N_f=2+1+1$ Lattice QCD: isospin-2 $\pi \pi$ scattering length
• Charmed baryon spectrum from lattice QCD near the physical point
• Lattice computation of the electromagnetic contributions to kaon and pion masses
• $⟨x⟩$ and $⟨x^2⟩$ of the pion PDF from Lattice QCD with $N_f=2+1+1$ dynamical quark flavours
• Lattice Calculation of the Intrinsic Soft Function and the Collins-Soper Kernel
• Status and Future Perspectives for Lattice Gauge Theory Calculations to the Exascale and Beyond
• Lattice QCD with Domain Decomposition on Intel Xeon Phi Co-Processors
• A Framework for Lattice QCD Calculations on GPUs
• $F_K/F_{\pi }$ from Möbius domain-wall fermions solved on gradient-flowed HISQ ensembles
• Hadron-Hadron Interactions from $N_f=2+1+1$ Lattice QCD: The $\rho$-resonance
• An accurate calculation of the nucleon axial charge with lattice QCD
• The $\rho$-resonance with physical pion mass from $N_f=2$ lattice QCD
• Coulomb, Landau and Maximally Abelian Gauge Fixing in Lattice QCD with Multi-GPUs
• On the generalised eigenvalue method and its relation to Prony and generalised pencil of function methods
• $\omega$ meson from lattice QCD
• Uncertainty Quantification in Lattice QCD Calculations for Nuclear Physics
• Computational Physics on Graphics Processing Units
• Multigrid for Staggered Lattice Fermions
• Ruling Out the Massless Up-Quark Solution to the Strong CP Problem by Computing the Topological Mass Contribution with Lattice QCD
• Scale setting the Möbius Domain Wall Fermion on gradient-flowed HISQ action using the omega baryon mass and the gradient-flow scales $t_0$ and $w_0$
• Möbius domain-wall fermions on gradient-flowed dynamical HISQ ensembles
• Design of an energy aware petaflops class high performance cluster based on power architecture
• Generating SU(Nc) pure gauge lattice QCD configurations on GPUs with CUDA
• Hadron-Hadron Interactions from $N_f=2+1+1$ Lattice QCD: $I=3/2$ $\pi K$ Scattering Length
• The axial charge of the triton from lattice QCD
• Hadron-Hadron Interactions from $N_f=2+1+1$ Lattice QCD: isospin-1 $KK$ scattering length
• Pushing Memory Bandwidth Limitations Through Efficient Implementations of Block-Krylov Space Solvers on GPUs
• Three-dimensional Imaging of Pion using Lattice QCD: Generalized Parton Distributions
• Light Cone Distribution Amplitude for the $\Lambda$ Baryon from Lattice QCD
• $\eta$-glueball mixing from $N_f=2$ lattice QCD
• ${\Lambda }_c{\Sigma }_c\pi$ coupling and ${\Sigma }_c\to {\Lambda }_c\pi$ decay in lattice QCD
• Lattice QCD on new chips: a community summary
• Lattice Gauge Theories at the Energy Frontier
• Energy Efficiency trends in HPC: what high-energy and astrophysicists need to know
• Conjugate gradient solvers on Intel Xeon Phi and NVIDIA GPUs
• Staggered Dslash Performance on Intel Xeon Phi Architecture
• Electromagnetic Structure of Spin-$\frac{1}{2}$ Doubly Charmed Baryons in Lattice QCD
• Disconnected quark loop contributions to nucleon observables using $N_f=2$ twisted clover fermions at the physical value of the light quark mass
• Evaluation of disconnected contributions using GPUs
• LatticeQCD using OpenCL
• Topological observables and $\theta$ dependence in high temperature QCD from lattice simulations
• Solving DWF Dirac Equation Using Multi-splitting Preconditioned Conjugate Gradient with Tensor Cores on NVIDIA GPUs
• Solving DWF Dirac Equation Using Multisplitting Preconditioned Conjugate Gradient
• Report of the Snowmass 2013 Computing Frontier working group on Lattice Field Theory -- Lattice field theory for the energy and intensity frontiers: Scientific goals and computing needs
• Implementation of the twisted mass fermion operator in the QUDA library
• Theoretical Perspective of Charm Physics
• Results on the disconnected contributions for hadron structure
• Calculation of the decay width of decuplet baryons
• A QUDA-branch to compute disconnected diagrams in GPUs
• Computation of disconnected contributions to nucleon observables
• CL2QCD - Lattice QCD based on OpenCL
• Lattice QCD package GWU-code and QUDA with HIP
• Gauge Field Generation on Large-Scale GPU-Enabled Systems
• Mixed Precision Solver Scalable to 16000 MPI Processes for Lattice Quantum Chromodynamics Simulations on the Oakforest-PACS System
• Impact of Dynamical Charm Quark and Mixed Action Effect on Light Hadron Masses and Decay Constants
• A heterogeneous many-core platform for experiments on scalable custom interconnects and management of fault and critical events, applied to many-process applications: Vol. II, 2012 technical report