The Outer Rim Simulation: A Path to Many-Core Supercomputers

Abstract

We describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225Gpc)^3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system like Mira, a many-core supercomputer, and how the simulation code, HACC, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5PB of data, making data curation and data handling a large challenge in of itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including DESI and eBOSS, as well as CMB experiments. A detailed catalog for the LSST DESC data challenges has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data.

10 pages, 10 figures. Submitted to ApJS. The Outer Rim data products can be accessed here: https://cosmology.alcf.anl.gov/

References in corpus (27)

• Seven-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Interpretation
• The cosmological simulation code GADGET-2
• Simulating the joint evolution of quasars, galaxies and their large-scale distribution
• Efficient Computation of CMB anisotropies in closed FRW models
• Mass function of dark matter halos
• Galaxy Clusters Discovered via the Sunyaev-Zel'dovich Effect in the 2500-square-degree SPT-SZ survey
• Galacticus: A Semi-Analytic Model of Galaxy Formation
• The clustering of massive galaxies at z~0.5 from the first semester of BOSS data
• Towards an accurate model of the redshift space clustering of halos in the quasilinear regime
• The Coyote Universe Extended: Precision Emulation of the Matter Power Spectrum
• HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures
• The mass of a halo
• Simulating the Universe with MICE: The abundance of massive clusters
• Mass Function Predictions Beyond LCDM
• The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey DR14 quasar sample: measurement of the growth rate of structure from the anisotropic correlation function between redshift 0.8 and 2.2
• The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey DR14 quasar sample: structure growth rate measurement from the anisotropic quasar power spectrum in the redshift range $0.8<z<2.2$
• The Mira-Titan Universe: Precision Predictions for Dark Energy Surveys
• Halo Profiles and the Concentration-Mass Relation for a ΛCDM Universe
• The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey DR14 quasar sample: anisotropic clustering analysis in configuration-space
• Tests of redshift-space distortions models in configuration space for the analysis of the BOSS final data release
• The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers
• Simulations of the Pairwise Kinematic Sunyaev-Zel'dovich Signal
• PICS: Simulations of Strong Gravitational Lensing in Galaxy Clusters
• Convergence of Galaxy Properties with Merger Tree Temporal Resolution
• HACC Cosmological Simulations: First Data Release
• Analyzing and Visualizing Cosmological Simulations with ParaView
• The importance of secondary halos for strong lensing in massive galaxy clusters across redshift

Cited by in corpus (74)

• Simulating cosmic structure formation with the GADGET-4 code
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: BAO and RSD measurements from anisotropic clustering analysis of the Quasar Sample in configuration space between redshift 0.8 and 2.2
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: measurement of the BAO and growth rate of structure of the luminous red galaxy sample from the anisotropic correlation function between redshifts 0.6 and 1
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: measurement of the BAO and growth rate of structure of the luminous red galaxy sample from the anisotropic power spectrum between redshifts 0.6 and 1.0
• The SPTpol Extended Cluster Survey
• Large-scale dark matter simulations
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Growth rate of structure measurement from anisotropic clustering analysis in configuration space between redshift 0.6 and 1.1 for the Emission Line Galaxy sample
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Large-scale Structure Catalogues and Measurement of the isotropic BAO between redshift 0.6 and 1.1 for the Emission Line Galaxy Sample
• The $\text{Abacus}$ Cosmological $N$-body Code
• Accurate initial conditions for cosmological N-body simulations: Minimizing truncation and discreteness errors
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: exploring the Halo Occupation Distribution model for Emission Line Galaxies
• The LSST DESC DC2 Simulated Sky Survey
• Galaxy Clusters Selected via the Sunyaev-Zel'dovich Effect in the SPTpol 100-Square-Degree Survey
• The MillenniumTNG Project: High-precision predictions for matter clustering and halo statistics
• Cosmological constraints from the power spectrum of eBOSS emission line galaxies
• Concentrations of Dark Haloes Emerge from Their Merger Histories
• Cosmological constraints without fingers of God
• Universal at last? The splashback mass function of dark matter halos
• The Completed SDSS-IV Extended Baryon Oscillation Spectroscopic Survey: N-body Mock Challenge for the Quasar Sample
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Cosmological implications from multi-tracer BAO analysis with galaxies and voids
• CARPool: fast, accurate computation of large-scale structure statistics by pairing costly and cheap cosmological simulations
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: N-body Mock Challenge for the eBOSS Emission Line Galaxy Sample
• Flybys, orbits, splashback: subhalos and the importance of the halo boundary
• The Completed SDSS-IV Extended Baryon Oscillation Spectroscopic Survey: N-body Mock Challenge for Galaxy Clustering Measurements
• Evidence for a high-z ISW signal from supervoids in the distribution of eBOSS quasars
• SNIa-Cosmology Analysis Results from Simulated LSST Images: from Difference Imaging to Constraints on Dark Energy
• The splashback radius of halos from particle dynamics: III. Halo catalogs, merger trees, and host-subhalo relations
• Strong lens modelling: comparing and combining Bayesian neural networks and parametric profile fitting
• A Parameter-Masked Mock Data Challenge for Beyond-Two-Point Galaxy Clustering Statistics
• A Joint Roman Space Telescope and Rubin Observatory Synthetic Wide-Field Imaging Survey
• The Strongest Cluster Lenses: An Analysis of the Relation Between Strong Gravitational Lensing Strength and the Physical Properties of Galaxy Clusters
• A Mock Catalog of Gravitationally Lensed Quasars for the LSST Survey
• The Last Journey. I. An Extreme-Scale Simulation on the Mira Supercomputer
• The impact of line-of-sight structures on measuring $H_0$ with strong lensing time-delays
• Growth-rate measurement with type-Ia supernovae using ZTF survey simulations
• A Composite Likelihood Approach for Inference under Photometric Redshift Uncertainty
• Reducing the Variance of Redshift Space Distortion Measurements from Mock Galaxy Catalogues with Different Lines of Sight
• Transitioning from Stage-III to Stage-IV: Cosmology from galaxy$\times$CMB lensing and shear$\times$CMB lensing
• Simulating image coaddition with the Nancy Grace Roman Space Telescope: I. Simulation methodology and general results
• The Sejong Suite: Cosmological Hydrodynamical Simulations with Massive Neutrinos, Dark Radiation, and Warm Dark Matter
• Farpoint: A High-Resolution Cosmology Simulation at the Gigaparsec Scale
• Euclid preparation. Modelling spectroscopic clustering on mildly nonlinear scales in beyond-$\Lambda$CDM models
• Simulating Hydrodynamics in Cosmology with CRK-HACC
• Galaxies and Halos on Graph Neural Networks: Deep Generative Modeling Scalar and Vector Quantities for Intrinsic Alignment
• Gaussian Process Classification for Galaxy Blend Identification in LSST
• OpenUniverse2024: A shared, simulated view of the sky for the next generation of cosmological surveys
• Improving Photometric Redshift Estimates with Training Sample Augmentation
• The Ultramarine Simulation: properties of dark matter haloes before redshift 5.5
• CUBE -- Towards an Optimal Scaling of Cosmological N-body Simulations
• Fully relativistic predictions in Horndeski gravity from standard Newtonian N-body simulations
• Efficient Mass Estimate at the Core of Strong Lensing Galaxy Clusters Using the Einstein Radius
• Chromatic Effects on the PSF and Shear Measurement for the Roman Space Telescope High-Latitude Wide Area Survey
• A method of weak lensing reconstruction through cosmic magnification with multi-band photometry information
• Recovered SN Ia rate from simulated LSST images
• Perturbation theory models for LSST-era galaxy clustering: tests with sub-percent mock catalog measurements in Fourier and configuration space
• Magrathea-Pathfinder: A 3D adaptive-mesh code for geodesic ray tracing in $N$-body simulations
• Core Mass Estimates in Strong Lensing Galaxy Clusters Using a Single-Halo Lens Model
• Improved Tomographic Binning of 3x2pt Lens Samples: Neural Network Classifiers and Optimal Bin Assignments
• Complementary Cosmological Simulations
• COmoving Computer Acceleration (COCA): $N$-body simulations in an emulated frame of reference
• Peculiar Velocity Reconstruction From Simulations and Observations Using Deep Learning Algorithms
• YOLO-CL cluster detection in the Rubin/LSST DC2 simulation
• Millimeter-wave observations of Euclid Deep Field South using the South Pole Telescope: A data release of temperature maps and catalogs
• Parameterization of Stochasticity in Galaxy Clustering and Reconstruction of Tomographic Matter Clustering
• Challenges in Cosmic Magnification Reconstruction by Magnification Response
• The Completed SDSS-IV Extended Baryon Oscillation Spectroscopic Survey: Growth rate of structure measurement from cosmic voids
• Delensing for Precision Cosmology: Optimizing Future CMB B-mode Surveys to Constrain r
• Improving the Accuracy of Halo Mass Based Statistics For Fast Approximate N-body Simulations
• The multi-dimensional halo assembly bias can be preserved when enhancing halo properties with HALOSCOPE
• Constraining Galaxy-Halo Connection Using Machine Learning
• Improved Methods for Estimating Peculiar Velocity Correlation Functions Using Volume Weighting
• Non-linear infusion of intrinsic alignment and source clustering: impact on non-Gaussian cosmic shear statistics
• Core Mass Estimates in Strong Lensing Galaxy Clusters: a Comparison Between Masses Obtained from Detailed Lens Models, Single-Halo Lens Models, and Einstein Radii
• Cosmological implications of the full shape of anisotropic clustering measurements in BOSS and eBOSS