nbodykit: an open-source, massively parallel toolkit for large-scale structure

Abstract

We present nbodykit, an open-source, massively parallel Python toolkit for analyzing large-scale structure (LSS) data. Using Python bindings of the Message Passing Interface (MPI), we provide parallel implementations of many commonly used algorithms in LSS. nbodykit is both an interactive and scalable piece of scientific software, performing well in a supercomputing environment while still taking advantage of the interactive tools provided by the Python ecosystem. Existing functionality includes estimators of the power spectrum, 2 and 3-point correlation functions, a Friends-of-Friends grouping algorithm, mock catalog creation via the halo occupation distribution technique, and approximate N-body simulations via the FastPM scheme. The package also provides a set of distributed data containers, insulated from the algorithms themselves, that enable nbodykit to provide a unified treatment of both simulation and observational data sets. nbodykit can be easily deployed in a high performance computing environment, overcoming some of the traditional difficulties of using Python on supercomputers. We provide performance benchmarks illustrating the scalability of the software. The modular, component-based approach of nbodykit allows researchers to easily build complex applications using its tools. The package is extensively documented at http://nbodykit.readthedocs.io, which also includes an interactive set of example recipes for new users to explore. As open-source software, we hope nbodykit provides a common framework for the community to use and develop in confronting the analysis challenges of future LSS surveys.

arXiv comment18 pages, 7 figures. Feedback very welcome. Code available at https://github.com/bccp/nbodykit and for documentation, see http://nbodykit.readthedocs.io

References in corpus (71)

• Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant
• Measurements of Omega and Lambda from 42 High-Redshift Supernovae
• Astropy: A Community Python Package for Astronomy
• The NumPy array: a structure for efficient numerical computation
• Planck 2015 results. XIII. Cosmological parameters
• Seven-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Interpretation
• Planck 2013 results. XVI. Cosmological parameters
• The cosmological simulation code GADGET-2
• Five-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Interpretation
• Nine-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Parameter Results
• Efficient Computation of CMB anisotropies in closed FRW models
• Detection of the Baryon Acoustic Peak in the Large-Scale Correlation Function of SDSS Luminous Red Galaxies
• The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample
• The Cosmic Linear Anisotropy Solving System (CLASS) II: Approximation schemes
• The 2dF Galaxy Redshift Survey: Power-spectrum analysis of the final dataset and cosmological implications
• Stable clustering, the halo model and nonlinear cosmological power spectra
• The Baryon Oscillation Spectroscopic Survey of SDSS-III
• GADGET: A code for collisionless and gasdynamical cosmological simulations
• The Rockstar Phase-Space Temporal Halo Finder and the Velocity Offsets of Cluster Cores
• A Multi-Code Analysis Toolkit for Astrophysical Simulation Data
• Revising the Halofit Model for the Nonlinear Matter Power Spectrum
• Massive neutrinos and cosmology
• Power Spectrum Analysis of Three-Dimensional Redshift Surveys
• Power Spectra for Cold Dark Matter and its Variants
• Baryon Acoustic Oscillations in the Lyα forest of BOSS DR11 quasars
• The SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Overview and Early Data
• Probing Dark Energy with Baryonic Acoustic Oscillations from Future Large Galaxy Redshift Surveys
• Galaxy Evolution from Halo Occupation Distribution Modeling of DEEP2 and SDSS Galaxy Clustering
• Quasar-Lyman $\alpha$ Forest Cross-Correlation from BOSS DR11 : Baryon Acoustic Oscillations
• Probing dark energy using baryonic oscillations in the galaxy power spectrum as a cosmological ruler
• Constraints on local primordial non-Gaussianity from large scale structure
• The Cosmological Constant is Back
• The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: Anisotropic galaxy clustering in Fourier-space
• Solving Large Scale Structure in Ten Easy Steps with COLA
• The Skewness of the Aperture Mass Statistic
• Uncorrelated Modes of the Nonlinear Power Spectrum
• Cosmic Complementarity: H_0 and Omega_m from Combining CMB Experiments and Redshift Surveys
• The mass function
• Accurate Estimators of Correlation Functions in Fourier Space
• FastPM: a new scheme for fast simulations of dark matter and halos
• Fast Estimators for Redshift-Space Clustering
• A Measurement of the Quadrupole Power Spectrum in the Clustering of the 2dF QSO Survey
• Mock galaxy catalogs using the quick particle mesh method
• Introducing Decorated HODs: modeling assembly bias in the galaxy-halo connection
• Measuring line-of-sight dependent Fourier-space clustering using FFTs
• Primordial non-Gaussianity from the large scale structure
• Detection of Baryon Acoustic Oscillation Features in the Large-Scale 3-Point Correlation Function of SDSS BOSS DR12 CMASS Galaxies
• Constraints onOmega from the IRAS Redshift Surveys
• Forward Modeling of Large-Scale Structure: An open-source approach with Halotools
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Signs of neutrino mass in current cosmological datasets
• A theoretical framework for combining techniques that probe the link between galaxies and dark matter
• A New Method to Correct for Fiber Collisions in Galaxy Two-Point Statistics
• Computing the Three-Point Correlation Function of Galaxies in $\mathcal{O}(N^2)$ Time
• Iterative initial condition reconstruction
• Beyond the plane-parallel approximation for redshift surveys
• An optimal FFT-based anisotropic power spectrum estimator
• Towards optimal parallel PM N-body codes: PMFAST
• Lagrangian perturbation theory at one loop order: successes, failures, and improvements
• Fast Algorithms and Efficient Statistics: N-point Correlation Functions
• Accelerating the 2-point and 3-point galaxy correlation functions using Fourier transforms
• Halo bias in Lagrangian Space: Estimators and theoretical predictions
• Generating Log-normal Mock Catalog of Galaxies in Redshift Space
• The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: constraining modified gravity
• Extending the modeling of the anisotropic galaxy power spectrum to $k=0.4h{\mathrm{Mpc}}^{-1}$
• An ideal mass assignment scheme for measuring the Power Spectrum with FFTs
• A Practical Computational Method for the Anisotropic Redshift-Space 3-Point Correlation Function
• Theoretical Systematics of Future Baryon Acoustic Oscillation Surveys
• Forecasts for the WFIRST High Latitude Survey using the BlueTides Simulation
• Imprint of DESI fiber assignment on the anisotropic power spectrum of emission line galaxies
• Reconstruction of halo power spectrum from redshift-space galaxy distribution: cylinder-grouping method and halo exclusion effect
• Galactos: Computing the Anisotropic 3-Point Correlation Function for 2 Billion Galaxies

Cited by in corpus (201)

• COLOSSUS: A python toolkit for cosmology, large-scale structure, and dark matter halos
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Baryon acoustic oscillations with Lyman-$\alpha$ forests
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: BAO and RSD measurements from anisotropic clustering analysis of the Quasar Sample in configuration space between redshift 0.8 and 2.2
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: BAO and RSD measurements from the anisotropic power spectrum of the Quasar sample between redshift 0.8 and 2.2
• DESI 2024 III: Baryon Acoustic Oscillations from Galaxies and Quasars
• Learning to Predict the Cosmological Structure Formation
• Large-scale dark matter simulations
• Accurate precision Cosmology with redshift unknown gravitational wave sources
• Redshift-Space Distortions in Lagrangian Perturbation Theory
• The Aemulus Project I: Numerical Simulations for Precision Cosmology
• Consistent Modeling of Velocity Statistics and Redshift-Space Distortions in One-Loop Perturbation Theory
• The $\text{Abacus}$ Cosmological $N$-body Code
• Redshift-weighted constraints on primordial non-Gaussianity from the clustering of the eBOSS DR14 quasars in Fourier space
• Modeling Biased Tracers at the Field Level
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: 1000 multi-tracer mock catalogues with redshift evolution and systematics for galaxies and quasars of the final data release
• Dark Photon Oscillations in Our Inhomogeneous Universe
• Constraints on the curvature of the Universe and dynamical dark energy from the Full-shape and BAO data
• Cosmological constraints from BOSS with analytic covariance matrices
• Accurate initial conditions for cosmological N-body simulations: Minimizing truncation and discreteness errors
• The mass-Peak Patch algorithm for fast generation of deep all-sky dark matter halo catalogues and its N-Body validation
• BOSS Correlation Function Analysis from the Effective Field Theory of Large-Scale Structure
• Testing the general theory of relativity using gravitational wave propagation from dark standard sirens
• The cosmology dependence of galaxy clustering and lensing from a hybrid $N$-body-perturbation theory model
• A complete FFT-based decomposition formalism for the redshift-space bispectrum
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: exploring the Halo Occupation Distribution model for Emission Line Galaxies
• Interpreting measurements of the anisotropic galaxy power spectrum
• UNIT project: Universe $N$-body simulations for the Investigation of Theoretical models from galaxy surveys
• Simulations and symmetries
• DESI 2024 II: Sample Definitions, Characteristics, and Two-point Clustering Statistics
• The MillenniumTNG Project: High-precision predictions for matter clustering and halo statistics
• Testing the theory of gravity with DESI: estimators, predictions and simulation requirements
• First Constraint on the Neutrino-Induced Phase Shift in the BAO Spectrum
• The BOSS bispectrum analysis at one loop from the Effective Field Theory of Large-Scale Structure
• Integral constraints in spectroscopic surveys
• DESI 2024 V: Full-Shape Galaxy Clustering from Galaxies and Quasars
• Updated constraints from the effective field theory analysis of BOSS power spectrum on Early Dark Energy
• Consistency of effective field theory analyses of the BOSS power spectrum
• Cosmological constraints from the power spectrum of eBOSS emission line galaxies
• The Impact of the Fiducial Cosmology Assumption on BAO Cosmological Parameter Inference
• Modeling Galaxies in Redshift Space at the Field Level
• Full-Shape Galaxy Power Spectra and the Curvature Tension
• Primordial Features from Linear to Nonlinear Scales
• Cosmology Without Windows: Quadratic Estimators for the Galaxy Power Spectrum
• The Completed SDSS-IV Extended Baryon Oscillation Spectroscopic Survey: N-body Mock Challenge for the Quasar Sample
• Cosmology Without Window Functions: Cubic Estimators for the Galaxy Bispectrum
• Simultaneous modelling of matter power spectrum and bispectrum in the presence of baryons
• Improving Galaxy Clustering Measurements with Deep Learning: analysis of the DECaLS DR7 data
• Baryonic effects on the matter bispectrum
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: N-body Mock Challenge for the eBOSS Emission Line Galaxy Sample
• Hefty enhancement of cosmological constraints from the DES Y1 data using a Hybrid Effective Field Theory approach to galaxy bias
• Separate Universe Simulations with IllustrisTNG: baryonic effects on power spectrum responses and higher-order statistics
• Limits on primordial non-Gaussianities from BOSS galaxy-clustering data
• A Hybrid Deep Learning Approach to Cosmological Constraints From Galaxy Redshift Surveys
• Modeling Dark Photon Oscillations in Our Inhomogeneous Universe
• Impact of Relativistic Effects on the Primordial Non-Gaussianity Signature in the Large-Scale Clustering of Quasars
• A fast particle-mesh simulation of non-linear cosmological structure formation with massive neutrinos
• AMBER: A Semi-numerical Abundance Matching Box for the Epoch of Reionization
• Cosmological constraints from the power spectrum of eBOSS quasars
• Detection of anisotropic galaxy assembly bias in BOSS DR12
• AI-assisted super-resolution cosmological simulations II: Halo substructures, velocities and higher order statistics
• Priors on red galaxy stochasticity from hybrid effective field theory
• Precision Tests of CO and [CII] Power Spectra Models against Simulated Intensity Maps
• Euclid: Forecasts from the void-lensing cross-correlation
• Non-perturbative probability distribution function for cosmological counts in cells
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• Galaxy Clustering Analysis with SimBIG and the Wavelet Scattering Transform
• The DESI $N$-body Simulation Project I: Testing the Robustness of Simulations for the DESI Dark Time Survey
• $\mathrm{S}\mathrm{IM}\mathrm{BIG}$: Mock Challenge for a Forward Modeling Approach to Galaxy Clustering
• A Parameter-Masked Mock Data Challenge for Beyond-Two-Point Galaxy Clustering Statistics
• Disconnected Covariance of 2-point Functions in Large-Scale Structure
• Baryon acoustic oscillations reconstruction using convolutional neural networks
• A fast semi-discrete optimal transport algorithm for a unique reconstruction of the early Universe
• Accurate predictions from small boxes: variance suppression via the Zel'dovich approximation
• Linear bias forecasts for emission line cosmological surveys
• Analytic Gaussian Covariance Matrices for Galaxy $N$-Point Correlation Functions
• Cosmological Information in Skew Spectra of Biased Tracers in Redshift Space
• Exploring KSZ velocity reconstruction with $N$-body simulations and the halo model
• Modeling Nearest Neighbor distributions of biased tracers using Hybrid Effective Field Theory
• $\text{matryoshka}$: Halo Model Emulator for the Galaxy Power Spectrum
• Multipole expansion for HI intensity mapping experiments: simulations and modelling
• Anisotropic halo assembly bias and redshift-space distortions
• Detecting the power spectrum turnover with HI intensity mapping
• The DESI $N$-body Simulation Project -- II. Suppressing sample variance with fast simulations
• Modeling HI at the field level
• Bayesian Inference of Initial Conditions from Non-Linear Cosmic Structures using Field-Level Emulators
• On the Cosmic Web Elongation in Fuzzy Dark Matter Cosmologies: Effects on Density Profiles, Shapes and Alignments of Halos
• Linear bias and halo occupation distribution of emission line galaxies from Nancy Grace Roman Space Telescope
• CSST Cosmological Emulator I: Matter Power Spectrum Emulation with one percent accuracy to k = 10 h/Mpc
• BOSS full-shape analysis from the EFTofLSS with exact time dependence
• Power spectrum multipole expansion for HI intensity mapping experiments: unbiased parameter estimation
• Joint velocity and density reconstruction of the Universe with nonlinear differentiable forward modeling
• A comparison between Shapefit compression and Full-Modelling method with PyBird for DESI 2024 and beyond
• PNG-UNITsims: Halo clustering response to primordial non-Gaussianities as a function of mass
• Investigating Cosmological GAN Emulators Using Latent Space Interpolation
• Fundamental Plane of BOSS galaxies: Correlations with galaxy properties, density field and impact on RSD measurements
• An Analytic Hybrid Halo + Perturbation Theory Model for Small-scale Correlators: Baryons, Halos, and Galaxies
• Tracer-Field Cross-Correlations with $k$-Nearest Neighbor Distributions
• A Faster Fourier Transform? Computing Small-Scale Power Spectra and Bispectra for Cosmological Simulations in $\mathcal{O}(N^2)$ Time
• Detection and estimation of the cosmic dipole with the Einstein Telescope and Cosmic Explorer
• Cleaning foregrounds from single-dish 21cm intensity maps with Kernel Principal Component Analysis
• Probing cosmic isotropy in the Local Universe
• Fast and realistic large-scale structure from machine-learning-augmented random field simulations
• Accurate Baryon Acoustic Oscillations reconstruction via semi-discrete optimal transport
• Generating mock galaxy catalogues for flux-limited samples like the DESI Bright Galaxy Survey
• RSD measurements from BOSS galaxy power spectrum using the halo perturbation theory model
• Validating the Galaxy and Quasar Catalog-Level Blinding Scheme for the DESI 2024 analysis
• COVMOS: a new Monte Carlo approach for galaxy clustering analysis
• Teaching neural networks to generate Fast Sunyaev Zel'dovich Maps
• Reducing the Variance of Redshift Space Distortion Measurements from Mock Galaxy Catalogues with Different Lines of Sight
• Super-sample covariance of the power spectrum, bispectrum, halos, voids, and their cross covariances
• The Kinematic Sunyaev-Zel'dovich Effect with ACT, DES, and BOSS: a Novel Hybrid Estimator
• Mapping Lyman-alpha forest three-dimensional large scale structure in real and redshift space
• Hybrid-basis inference for large-scale galaxy clustering: combining spherical and Cartesian Fourier analyses
• Beyond Yamamoto: Anisotropic Power Spectra and Correlation Functions with Pairwise Lines-of-Sight
• Halo assembly bias from a deep learning model of halo formation
• The Effect of Systematic Redshift Biases in BAO Cosmology
• Building a digital twin of a luminous red galaxy spectroscopic survey: galaxy properties and clustering covariance
• The Global Environment of Small Galaxy Systems
• Comparing weak lensing peak counts in baryonic correction models to hydrodynamical simulations
• Edges and Endpoints in 21-cm Observations from Resonant Photon Production
• Line confusion in spectroscopic surveys and its possible effects: Shifts in Baryon Acoustic Oscillations position
• The impact and response of minihalos and the inter-halo medium on cosmic reionization
• Extracting high-order cosmological information in galaxy surveys with power spectra
• A multi-tracer empirically-driven approach to line-intensity mapping lightcones
• Flow-Based Likelihoods for Non-Gaussian Inference
• AI-assisted super-resolution cosmological simulations III: Time evolution
• Clustering in the Simulated H$\alpha$ Galaxy Redshift Survey from Nancy Grace Roman Space Telescope
• Sensitivity tests of cosmic velocity fields to massive neutrinos
• Principal Component Analysis of Galaxy Clustering in Hyperspace of Galaxy Properties
• Large Scale Structure Reconstruction with Short-Wavelength Modes
• The BAHAMAS project: Evaluating the accuracy of the halo model in predicting the non-linear matter power spectrum
• Revisiting Vainshtein Screening for fast N-body simulations
• Looking for a twist: probing the cosmological gravitomagnetic effect via weak lensing-kSZ cross correlations
• High-fidelity reproduction of central galaxy joint distributions with Neural Networks
• Three-point intrinsic alignments of dark matter halos in the IllustrisTNG simulation
• Measuring the BAO peak position with different galaxy selections
• Setting the Stage: Structures from Gaussian Random Fields
• On the Fast Random Sampling and Other Properties of the Three Point Correlation Function in Galaxy Surveys
• Fast generation of mock galaxy catalogues with COLA
• Testing the SZ-based tomographic approach to the thermal history of the universe with pressure-density cross-correlations: Insights from the Magneticum simulation
• The halo bispectrum as a sensitive probe of massive neutrinos and baryon physics
• Searching for dilaton fields in the Ly$\alpha$ forest
• How do baryonic effects on the cosmic matter distribution vary with scale and local density environment?
• The HalfDome Multi-Survey Cosmological Simulations: N-body Simulations
• Validating galaxy clustering models with Fixed & Paired and Matched-ICs simulations: application to Primordial Non-Gaussianities
• Tree-less 3d Friends-of-Friends using Spatial Hashing
• A Novel Estimator for the Equation of State of the IGM by Ly$\alpha$ Forest Tomography
• Constraining the growth rate of structure with phase correlations
• Modeling Post-Reionization HI Distributions in Fuzzy Dark Matter Cosmologies Using Conditional Normalizing Flows
• Triumvirate: A Python/C++ package for three-point clustering measurements
• HI HOD -- I. The Halo Occupation Distribution of HI Galaxies
• Perturbation theory models for LSST-era galaxy clustering: tests with sub-percent mock catalog measurements in Fourier and configuration space
• Baryon acoustic scale at $z_{\eff} = 0.166$ with the SDSS blue galaxies
• Baryon acoustic oscillations from HI intensity mapping: the importance of cross-correlations in the monopole and quadrupole
• Cosmic Tidal Reconstruction with Halo Fields
• ConKer: evaluating isotropic correlations of arbitrary order
• MF-Box: Multi-fidelity and multi-scale emulation for the matter power spectrum
• A New Probe of the High-z BAO scale: BAO tomography With CMB $\times$ LIM-Nulling Convergence
• Psi-GAN: A power-spectrum-informed generative adversarial network for the emulation of large-scale structure maps across cosmologies and redshifts
• Cosmological constraints from the redshift dependence of the Alcock-Paczynski effect: Fourier space analysis
• Bispectrum constraints on Primordial non-Gaussianities with the eBOSS DR16 quasars
• Inferring Cosmological Parameters on SDSS via Domain-Generalized Neural Networks and Lightcone Simulations
• Ray-tracing log-normal simulation for weak gravitational lensing: application to the cross-correlation with galaxies
• Impact of cosmological signatures in two-point statistics beyond the linear regime
• Nonlinear cosmological structure with ultralight bosons via modified gravity
• Anisotropic effective redshift and evolving clustering amplitude
• Assessing theoretical uncertainties for cosmological constraints from weak lensing surveys
• DESI Mock Challenge: Constructing DESI galaxy catalogues based on FastPM simulations
• Cosmological inference including massive neutrinos from the matter power spectrum: biases induced by uncertainties in the covariance matrix
• Subhalo abundance matching using progenitor mass at varying redshift: Two modes of stellar mass growth imprinted into the Subaru HSC galaxy clustering
• Proca-stinated Cosmology II: Matter, Halo, and Lensing Statistics in the vector Galileon
• Intrinsic correlations of galaxy sizes in a hydrodynamical cosmological simulation
• An Optimized Ly$\alpha$ Forest Inversion Tool Based on a Quantitative Comparison of Existing Reconstruction Methods
• Constraining $f_{NL}$ using the Large-Scale Modulation of Small-Scale Statistics
• How do uncertainties in galaxy formation physics impact field-level galaxy bias?
• Angular Correlation Function from sample covariance with BOSS and eBOSS LRG
• Turbulence revealed by wavelet transform: power spectrum and intermittency for the velocity field of the cosmic baryonic fluid
• Self-calibrating BAO measurements in the presence of Small Displacement Interlopers
• Baryon Acoustic Oscillations analyses with Density-Split Statistics
• Parameterization of Stochasticity in Galaxy Clustering and Reconstruction of Tomographic Matter Clustering
• An Algorithm to locate the centers of Baryon Acoustic Oscillations
• The Cosmic Mach Number as an Environment Measure for the Underlying Dark Matter Density Field
• Phenomenological power spectrum models for H$\alpha$ emission line galaxies from the Nancy Grace Roman Space Telescope
• Deep learning reconstruction of the large scale structure of the Universe from luminosity distance observations
• Pair Counting without Binning -- A New Approach to Correlation Functions in Clustering Statistics
• A comparison of shrinkage estimators of the cosmological precision matrix
• Mapping the cosmic expansion history from LIGO-Virgo-KAGRA in synergy with DESI and SPHEREx
• Baryon Acoustic Oscillations from galaxy surveys
• Impact of a Rapid Diluted Energy Density on the halo mass function
• Super sample covariance and the volume scaling of galaxy survey covariance matrices
• Elucidating the impact of massive neutrinos on halo assembly bias
• Accurate Power Spectrum Estimation toward Nyquist limit
• Mimicking the halo-galaxy connection using machine learning
• The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey quasar sample: Testing observational systematics on the Baryon Acoustic Oscillation measurement
• Lagrangian Perturbation Theory for Biased Tracers: Significance of the Number Conservation
• Ringing the universe with cosmic emptiness: void properties through a combined analysis of stacked weak gravitational and Doppler lensing
• Fast Sampling of Cosmological Initial Conditions with Gaussian Neural Posterior Estimation
• Unclustered tracers remain unclustered: the lack of primordial non-Gaussianity response of bias-zero tracers
• The frame-dragging vector potential on galaxy scales from Dark-Matter-only Newtonian $N$-body simulations
• Cosmic voids and BAO with relative baryon-CDM perturbations
• Cosmic curl -- Features and convergence of the vorticity power spectrum in $N$-body simulations