Corrfunc --- A Suite of Blazing Fast Correlation Functions on the CPU

Abstract

The two-point correlation function (2PCF) is the most widely used tool for quantifying the spatial distribution of galaxies. Since the distribution of galaxies is determined by galaxy formation physics as well as the underlying cosmology, fitting an observed correlation function yields valuable insights into both. The calculation for a 2PCF involves computing pair-wise separations and consequently, the computing time scales quadratically with the number of galaxies. The next-generation galaxy surveys are slated to observe many millions of galaxies, and computing the 2PCF for such surveys would be prohibitively time-consuming. Additionally, modern modelling techniques require the 2PCF to be calculated thousands of times on simulated galaxy catalogues of {\em at least} equal size to the data and would be completely unfeasible for the next generation surveys. Thus, calculating the 2PCF forms a substantial bottleneck in improving our understanding of the fundamental physics of the universe, and we need high-performance software to compute the correlation function. In this paper, we present Corrfunc --- a suite of highly optimised, OpenMP parallel clustering codes. The improved performance of Corrfunc arises from both efficient algorithms as well as software design that suits the underlying hardware of modern CPUs. Corrfunc can compute a wide range of 2-D and 3-D correlation functions in either simulation (Cartesian) space or on-sky coordinates. Corrfunc runs efficiently in both single- and multi-threaded modes and can compute a typical 2-point projected correlation function ($w_p(r_p)$) for ~1 million galaxies within a few seconds on a single thread. Corrfunc is designed to be both user-friendly and fast and is publicly available at https://github.com/manodeep/Corrfunc.

arXiv commentAccepted for publication to MNRAS

References in corpus (39)

• Scikit-learn: Machine Learning in Python
• The NumPy array: a structure for efficient numerical computation
• Detection of the Baryon Acoustic Peak in the Large-Scale Correlation Function of SDSS Luminous Red Galaxies
• The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample
• The 6dF Galaxy Survey: Baryon Acoustic Oscillations and the Local Hubble Constant
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Baryon Acoustic Oscillations in the Data Release 10 and 11 galaxy samples
• Baryon Acoustic Oscillations in the Sloan Digital Sky Survey Data Release 7 Galaxy Sample
• Sloan Digital Sky Survey IV: Mapping the Milky Way, Nearby Galaxies, and the Distant Universe
• Cosmological Constraints from the SDSS Luminous Red Galaxies
• Constraints on the relationship between stellar mass and halo mass at low and high redshift
• KiDS-450: Cosmological parameter constraints from tomographic weak gravitational lensing
• The WiggleZ Dark Energy Survey: mapping the distance-redshift relation with baryon acoustic oscillations
• Euclid Definition Study Report
• The Luminosity and Color Dependence of the Galaxy Correlation Function
• The Connection between Galaxies and their Dark Matter Halos
• Galaxy Clustering in the Completed SDSS Redshift Survey: The Dependence on Color and Luminosity
• Modeling Luminosity-Dependent Galaxy Clustering Through Cosmic Time
• The age dependence of halo clustering
• Constraining Galaxy Formation and Cosmology with the Conditional Luminosity Function of Galaxies
• The Dependence of Halo Clustering on Halo Formation History, Concentration, and Occupation
• New constraints on the evolution of the stellar-to-dark matter connection: a combined analysis of galaxy-galaxy lensing, clustering, and stellar mass functions from z=0.2 to z=1
• The 2dF Galaxy Redshift Survey: The dependence of galaxy clustering on luminosity and spectral type
• The DESI Experiment, a whitepaper for Snowmass 2013
• The Skewness of the Aperture Mass Statistic
• Studying Reionization with Ly-alpha Emitters
• Cosmological Constraints from a Combination of Galaxy Clustering and Lensing -- I. Theoretical Framework
• Systematic Identification of LAEs for Visible Exploration and Reionization Research Using Subaru HSC (SILVERRUSH). I. Program Strategy and Clustering Properties of ~2,000 Lya Emitters at z=6-7 over the 0.3-0.5 Gpc${}^2$ Survey Area
• The Dark Side of Galaxy Color
• Mapping stellar content to dark matter halos using galaxy clustering and galaxy-galaxy lensing in the SDSS DR7
• Galaxy clustering in the CFHTLS-Wide: the changing relationship between galaxies and haloes since z ~ 1.2
• Clustering in the 1.2 Jy IRAS Galaxy Redshift Survey I: The Redshift and Real Space Correlation Functions
• Forward Modeling of Large-Scale Structure: An open-source approach with Halotools
• MLPACK: A Scalable C++ Machine Learning Library
• Spatial Clustering of Dark Matter Halos: Secondary Bias, Neighbor Bias, and the Influence of Massive Neighbors on Halo Properties
• Dark Energy Survey Year 1 Results: Cross-Correlation Redshifts - Methods and Systematics Characterization
• Improving Initial Conditions for Cosmological $N$-Body Simulations
• CUTE solutions for two-point correlation functions from large cosmological datasets
• Corrfunc: Blazing fast correlation functions with AVX512F SIMD Intrinsics
• Pairwise velocities in the "Running FLRW" cosmological model

Cited by in corpus (129)

• DESI DR2 Results II: Measurements of Baryon Acoustic Oscillations and Cosmological Constraints
• AbacusSummit: A Massive Set of High-Accuracy, High-Resolution $N$-Body Simulations
• Large-scale dark matter simulations
• The $\text{Abacus}$ Cosmological $N$-body Code
• Nearest Neighbor distributions: new statistical measures for cosmological clustering
• AbacusHOD: A highly efficient extended multi-tracer HOD framework and its application to BOSS and eBOSS data
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: exploring the Halo Occupation Distribution model for Emission Line Galaxies
• Dissecting and Modelling Galaxy Assembly Bias
• DESI 2024 II: Sample Definitions, Characteristics, and Two-point Clustering Statistics
• Cosmological Analysis of Three-Dimensional BOSS Galaxy Clustering and Planck CMB Lensing Cross Correlations via Lagrangian Perturbation Theory
• Stringent ${\sigma }_8$ constraints from small-scale galaxy clustering using a hybrid MCMC+emulator framework
• Evidence for galaxy assembly bias in BOSS CMASS redshift-space galaxy correlation function
• Semi-analytic forecasts for JWST -- VI. Simulated lightcones and galaxy clustering predictions
• The galaxy-halo connection of emission-line galaxies in IllustrisTNG
• The Aemulus Project V: Cosmological constraint from small-scale clustering of BOSS galaxies
• Cosmological cross-correlations and nearest neighbor distributions
• Euclid. V. The Flagship galaxy mock catalogue: a comprehensive simulation for the Euclid mission
• The MillenniumTNG Project: An improved two-halo model for the galaxy-halo connection of red and blue galaxies
• First Detection of the BAO Signal from Early DESI Data
• Do model emission line galaxies live in filaments at z~1?
• Deep Realistic Extragalactic Model (DREaM) Galaxy Catalogs: Predictions for a Roman Ultra-Deep Field
• Testing the Accuracy of Halo Occupation Distribution Modelling using Hydrodynamic Simulations
• The MillenniumTNG Project: Refining the one-halo model of red and blue galaxies at different redshifts
• Precise Cosmological Constraints from BOSS Galaxy Clustering with a Simulation-Based Emulator of the Wavelet Scattering Transform
• CoLoRe: fast cosmological realisations over large volumes with multiple tracers
• Semi-analytic forecasts for Roman -- the beginning of a new era of deep-wide galaxy surveys
• The AGN-galaxy-halo connection: The distribution of AGN host halo masses to z=2.5
• The physical drivers of the atomic hydrogen-halo mass relation
• Mitigation of DESI fiber assignment incompleteness effect on two-point clustering with small angular scale truncated estimators
• Semi-analytical covariance matrices for two-point correlation function for DESI 2024 data
• The cosmic web of X-ray active galactic nuclei seen through the eROSITA Final Equatorial Depth Survey (eFEDS)
• The impact of baryonic physics on the abundance, clustering, and concentration of halos
• Optimal Reconstruction of Baryon Acoustic Oscillations for DESI 2024
• Combination of cluster number counts and two-point correlations: Validation on Mock Dark Energy Survey
• Testing General Relativity on cosmological scales at redshift z ~ 1.5 with quasar and CMB lensing
• A unified model for the clustering of quasars and galaxies at $z\approx 6$
• The DESI $N$-body Simulation Project -- II. Suppressing sample variance with fast simulations
• Quantifying resolution in cosmological N-body simulations using self-similarity
• Euclid: Cosmology forecasts from the void-galaxy cross-correlation function with reconstruction
• A First Detection of the Connected 4-Point Correlation Function of Galaxies Using the BOSS CMASS Sample
• Constructing high-fidelity halo merger trees in AbacusSummit
• The formation of supermassive black holes from Population III.1 seeds. II. Evolution to the local universe
• The Aemulus Project VI: Emulation of beyond-standard galaxy clustering statistics to improve cosmological constraints
• Host Dark Matter Halos of WISE-selected Obscured & Unobscured Quasars: Evidence for Evolution
• The assembly bias of emission line galaxies
• The void-galaxy cross-correlation function with massive neutrinos and modified gravity
• An Analytic Hybrid Halo + Perturbation Theory Model for Small-scale Correlators: Baryons, Halos, and Galaxies
• Using angular two-point correlations to self-calibrate the photometric redshift distributions of DECaLS DR9
• Testing dark matter halo properties using self-similarity
• DESI Legacy Imaging Surveys Data Release 9: Cosmological Constraints from Galaxy Clustering and Weak Lensing using the Minimal Bias Model
• ELG Spectroscopic Systematics Analysis of the DESI Data Release 1
• Validating the Galaxy and Quasar Catalog-Level Blinding Scheme for the DESI 2024 analysis
• A measurement of the scale of homogeneity in the Early Universe
• Full forward model of galaxy clustering statistics with AbacusSummit lightcones
• Beyond Yamamoto: Anisotropic Power Spectra and Correlation Functions with Pairwise Lines-of-Sight
• Revisiting the Extreme Clustering of $z\approx 4$ Quasars with Large Volume Cosmological Simulations
• Good and Proper: Self-similarity of N-body Simulations with Proper Force Softening
• Differentiable Predictions for Large Scale Structure with SHAMNet
• Host Dark Matter Halos of SDSS Red and Blue Quasars: No Significant Difference in Large-scale Environment
• Large Scale Overdensity of Lyman Break Galaxies Around the z=6.3 Ultraluminous Quasar J0100+2802
• 2D k-th nearest neighbor statistics: a highly informative probe of galaxy clustering
• Catalogues of voids as antihalos in the local Universe
• Halo Occupation Distribution of Emission Line Galaxies: fitting method with Gaussian Processes
• Galaxy Clustering in the Mira-Titan Universe I: Emulators for the redshift space galaxy correlation function and galaxy-galaxy lensing
• Fast Correlation Function Calculator -- A high-performance pair counting toolkit
• High redshift LBGs from deep broadband imaging for future spectroscopic surveys
• Spatial and Kinematic Clustering of Stars in the Galactic Disk
• Tracking the orbit of unresolved subhalos for semi-analytic models
• Thermal and Reionisation History within a Large-Volume Semi-Analytic Galaxy Formation Simulation
• The DESI One-Percent Survey: Modelling the clustering and halo occupation of all four DESI tracers with Uchuu
• Rapid Simulations of Halo and Subhalo Clustering
• Toward Accurate Modeling of Galaxy Clustering on Small Scales: Constraining the Galaxy-Halo Connection with Optimal Statistics
• Galaxy and Mass Assembly (GAMA): the clustering of galaxy groups
• Detecting Baryon Acoustic Oscillations with third generation gravitational wave observatories
• Toward Accurate Modeling of Galaxy Clustering on Small Scales: Halo Model Extensions and Lingering Tension
• A New Stellar Mass Proxy for Subhalo Abundance Matching
• Towards an optimal marked correlation function analysis for the detection of modified gravity
• PAC in DESI. I. Galaxy Stellar Mass Function into the $10^6{\mathrm{M}}_{\odot }$ Frontier
• Constraining galaxy-halo connection with high-order statistics
• The effect of local universe constraints on halo abundance and clustering
• Fast analytical calculation of the random pair counts for realistic survey geometry
• Fast Forward Modelling of Galaxy Spatial and Statistical Distributions
• The feasibility of constraining DM interactions with high-redshift observations by JWST
• Halo mass function in scale invariant models
• Two-point statistics without bins: A continuous-function generalization of the correlation function estimator for large-scale structure
• An efficient hybrid method to produce high resolution large volume dark matter simulations for semi-analytic models of reionisation
• HI HOD -- I. The Halo Occupation Distribution of HI Galaxies
• Clustering of emission line galaxies with IllustrisTNG I.: fundamental properties and halo occupation distribution
• Small scale clustering of BOSS galaxies: dependence on luminosity, color, age, stellar mass, specific star formation rate and other properties
• Star-forming and Quiescent Central Galaxies Cluster Similarly: Implications for the Galaxy-Halo Connection
• A two-phase model of galaxy formation: III. The formation of globular clusters
• Subhalo abundance matching using progenitor mass at varying redshift: Two modes of stellar mass growth imprinted into the Subaru HSC galaxy clustering
• Cosmological constraints from galaxy clustering and galaxy-galaxy lensing with extended SubHalo Abundance Matching
• The role of scatter and satellites in shaping the large-scale clustering of X-ray AGN as a function of host galaxy stellar mass
• Cross-correlation between soft X-rays and galaxies A new benchmark for galaxy evolution models
• Probing the large-scale structure of the universe through gravitational-wave observations
• Impact of lensing magnification on the analysis of galaxy clustering in redshift space
• Unraveling emission line galaxy conformity at z~1 with DESI early data
• The power spectrum of galaxies from large to small scales: a line-intensity mapping perspective
• Galaxy assembly bias and large-scale distribution: a comparison between IllustrisTNG and a semi-analytic model
• Galaxy Assembly and Evolution in the P-Millennium simulation: galaxy clustering
• The 4MOST-Cosmology Redshift Survey: Target Selection of Bright Galaxies and Luminous Red Galaxies
• Angular Correlation Function from sample covariance with BOSS and eBOSS LRG
• Improving the accuracy of estimators for the two-point correlation function
• CLIMBER: Galaxy-Halo Connection Constraints from Next-Generation Surveys
• Quasar Pairs as Large--Scale Structure tracers
• Baryon Acoustic Oscillations analyses with Density-Split Statistics
• Galaxy Correlation Function and Local Density from Photometric Redshifts Using the Stochastic Order Redshift Technique (SORT)
• Convergence of halo statistics: code comparison between Rockstar and CompaSO using scale-free simulations
• Euclid preparation. 3-dimensional galaxy clustering in configuration space. Part I. 2-point correlation function estimation
• Impact of projection-induced optical selection bias on the weak lensing mass calibration of galaxy clusters
• DESI Emission-line Galaxies: Clustering Dependence on Stellar Mass and [OII] Luminosity
• Assessing the connection between galactic conformity and assembly-type bias
• Studying baryon acoustic oscillations using photometric redshifts from the DESI Legacy Imaging survey DR9
• Pair Counting without Binning -- A New Approach to Correlation Functions in Clustering Statistics
• The clustering of C IV and Si IV at the end of reionisation: A perspective from the E-XQR-30 survey
• Using two point correlation functions to understand the assembly histories of Milky Way-like galaxies
• Constraining Galaxy-Halo Connection Using Machine Learning
• The mass and redshift dependence of halo star clustering
• Influence of photometric galaxies redshift distribution in BAO estimation
• Towards an Observational Detection of Halo Spin Bias using Spin-Orbit Coherence
• Photometric Objects Around Cosmic Webs (PAC). VI. High Satellite Fraction of Quasars
• A theoretical approach to density-split clustering
• On the dependence of galaxy assembly bias on the selection criteria, number density, and redshift of galaxy samples
• Marked statistics across the cosmic web: Environmental dependent clustering in modified gravity simulations
• Web-Halo Model (WHM): Accurate non-linear matter power spectrum predictions without free parameters
• FuGa3D: Fast full-sky analysis of Galaxy catalogs in 3D
• Clustering and physical properties of AGN and Star-Forming Galaxies at fixed stellar mass: does assembly bias have a role in AGN activity?
• An emulation-based model for the projected correlation function