Nearest Neighbor distributions: new statistical measures for cosmological clustering

Abstract

The use of summary statistics beyond the two-point correlation function to analyze the non-Gaussian clustering on small scales is an active field of research in cosmology. In this paper, we explore a set of new summary statistics -- the $k$-Nearest Neighbor Cumulative Distribution Functions ($k\mathrm{NN}$-$\mathrm{CDF}$). This is the empirical cumulative distribution function of distances from a set of volume-filling, Poisson distributed random points to the $k$-nearest data points, and is sensitive to all connected $N$-point correlations in the data. The $k\mathrm{NN}$-$\mathrm{CDF}$ can be used to measure counts in cell, void probability distributions and higher $N$-point correlation functions, all using the same formalism exploiting fast searches with spatial tree data structures. We demonstrate how it can be computed efficiently from various data sets - both discrete points, and the generalization for continuous fields. We use data from a large suite of $N$-body simulations to explore the sensitivity of this new statistic to various cosmological parameters, compared to the two-point correlation function, while using the same range of scales. We demonstrate that the use of $k\mathrm{NN}$-$\mathrm{CDF}$ improves the constraints on the cosmological parameters by more than a factor of $2$ when applied to the clustering of dark matter in the range of scales between $10h^{-1}\mathrm{Mpc}$ and $40h^{-1}\mathrm{Mpc}$. We also show that relative improvement is even greater when applied on the same scales to the clustering of halos in the simulations at a fixed number density, both in real space, as well as in redshift space. Since the $k\mathrm{NN}$-$\mathrm{CDF}$ are sensitive to all higher order connected correlation functions in the data, the gains over traditional two-point analyses are expected to grow as progressively smaller scales are included in the analysis of cosmological data.

Minor changes. 1 figure added. Matches published version in MNRAS

References in corpus (23)

• Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
• The imprints of primordial non-gaussianities on large-scale structure: scale dependent bias and abundance of virialized objects
• The Effective Field Theory of Cosmological Large Scale Structures
• Improving Cosmological Distance Measurements by Reconstruction of the Baryon Acoustic Peak
• Cosmology and the Bispectrum
• Corrfunc --- A Suite of Blazing Fast Correlation Functions on the CPU
• Planck 2018 results. IX. Constraints on primordial non-Gaussianity
• Reconstructing Baryon Oscillations: A Lagrangian Theory Perspective
• The Mira-Titan Universe II: Matter Power Spectrum Emulation
• Simulating nonlinear cosmological structure formation with massive neutrinos
• Emulating the CFHTLenS Weak Lensing data: Cosmological Constraints from moments and Minkowski functionals
• Using the Marked Power Spectrum to Detect the Signature of Neutrinos in Large-Scale Structure
• A marked correlation function for constraining modified gravity models
• Analytic Minkowski Functionals of the Cosmic Microwave Background: Second-order Non-Gaussianity with Bispectrum and Trispectrum
• Simulations and symmetries
• Lagrangian perturbation theory at one loop order: successes, failures, and improvements
• On the inadequacy of N-point correlation functions to describe nonlinear cosmological fields: explicit examples and connection to simulations
• Cosmological constraints with weak lensing peak counts and second-order statistics in a large-field survey
• Perturbation theory and excursion set estimates of the probability distribution function of dark matter, and a method for reconstructing the initial distribution function
• Voronoi volume function: A new probe of cosmology and galaxy evolution
• Probing Galaxy Assembly Bias in BOSS Galaxies Using Void Probabilities
• Moving around the cosmological parameter space: a nonlinear power spectrum reconstruction based on high-resolution cosmic responses
• Galaxy Bias and ${\sigma }_8$ from Counts in Cells from the SDSS Main Sample

Cited by in corpus (71)

• The CAMELS project: Cosmology and Astrophysics with MachinE Learning Simulations
• Large-scale dark matter simulations
• Towards an Optimal Estimation of Cosmological Parameters with the Wavelet Scattering Transform
• Constraining $M_{\nu }$ with the Bispectrum II: The Total Information Content of the Galaxy Bispectrum
• Detecting Neutrino Mass by Combining Matter Clustering, Halos, and Voids
• The GIGANTES dataset: precision cosmology from voids in the machine learning era
• Stringent ${\sigma }_8$ constraints from small-scale galaxy clustering using a hybrid MCMC+emulator framework
• The CAMELS project: public data release
• Going Beyond the Galaxy Power Spectrum: an Analysis of BOSS Data with Wavelet Scattering Transforms
• A flexible subhalo abundance matching model for galaxy clustering in redshift space
• Learning cosmology and clustering with cosmic graphs
• Cosmological cross-correlations and nearest neighbor distributions
• Robust Field-level Likelihood-free Inference with Galaxies
• Cosmology with one galaxy?
• The MillenniumTNG Project: An improved two-halo model for the galaxy-halo connection of red and blue galaxies
• Aemulus $\nu$: Precise Predictions for Matter and Biased Tracer Power Spectra in the Presence of Neutrinos
• Precise Cosmological Constraints from BOSS Galaxy Clustering with a Simulation-Based Emulator of the Wavelet Scattering Transform
• A Parameter-Masked Mock Data Challenge for Beyond-Two-Point Galaxy Clustering Statistics
• The MillenniumTNG Project: Inferring cosmology from galaxy clustering with accelerated N-body scaling and subhalo abundance matching
• NECOLA: Towards a Universal Field-level Cosmological Emulator
• Modeling Nearest Neighbor distributions of biased tracers using Hybrid Effective Field Theory
• Evidence of Galaxy Assembly Bias in SDSS DR7 Galaxy Samples from Count Statistics
• Joint velocity and density reconstruction of the Universe with nonlinear differentiable forward modeling
• Observational Measures of Halo Properties Beyond Mass
• Fisher Forecasts for Primordial non-Gaussianity from Persistent Homology
• Neural networks as optimal estimators to marginalize over baryonic effects
• The Aemulus Project VI: Emulation of beyond-standard galaxy clustering statistics to improve cosmological constraints
• Tracer-Field Cross-Correlations with $k$-Nearest Neighbor Distributions
• How do Primordial Black Holes change the Halo Mass Function and Structure?
• Fast and Accurate Non-Linear Predictions of Universes with Deep Learning
• Main Sequence Scatter is Real: The Joint Dependence of Galaxy Clustering on Star Formation and Stellar Mass
• Non-parametric reconstruction of cosmological observables using Gaussian Processes Regression
• Cosmology with Persistent Homology: a Fisher Forecast
• Cosmological Probes of Structure Growth and Tests of Gravity
• Full forward model of galaxy clustering statistics with AbacusSummit lightcones
• Super-sample covariance of the power spectrum, bispectrum, halos, voids, and their cross covariances
• Detection of spatial clustering in the 1000 richest SDSS DR8 redMaPPer clusters with Nearest Neighbor distributions
• Quantification of high dimensional non-Gaussianities and its implication to Fisher analysis in cosmology
• Differentiable Predictions for Large Scale Structure with SHAMNet
• 2D k-th nearest neighbor statistics: a highly informative probe of galaxy clustering
• Impact of tidal environment on galaxy clustering in GAMA
• Simultaneous Dependence of Matter Clustering on Scale and Environment
• Towards unveiling the large-scale nature of gravity with the wavelet scattering transform
• Validating the clustering predictions of empirical models with the FLAMINGO simulations
• Hitting the mark: Optimising Marked Power Spectra for Cosmology
• Field-level simulation-based inference with galaxy catalogs: the impact of systematic effects
• The catalog-to-cosmology framework for weak lensing and galaxy clustering for LSST
• Topological data analysis reveals differences between simulated galaxies and dark matter haloes
• Inpainting hydrodynamical maps with deep learning
• An Analytically Tractable Marked Power Spectrum
• Cosmology from Point Clouds with Dark Matter Halos from the Quijote Simulations
• High-energy Neutrino Source Cross-correlations with Nearest-neighbor Distributions
• Self-Similarity of $k$-Nearest Neighbor Distributions in Scale-Free Simulations
• Probing Patchy Reionization with the Void Probability Function of Lyman-$\alpha$ Emitters
• Geometric Interpretations of the $k$-Nearest Neighbour Distributions
• Bounds on galaxy stochasticity from halo occupation distribution modeling
• Cosmology with Persistent Homology: Parameter Inference via Machine Learning
• One-point Statistics in various cosmic environments in the presence of massive neutrinos
• A Conditional Abundance Matching Method of Extending Simulated Halo Merger Trees to Resolve Low-Mass Progenitors and Sub-halos
• Clustering of dark matter in the cosmic web as a probe of massive neutrinos
• A theoretical approach to density-split clustering
• Insights into the dependence of galaxy properties on the environment with explainable machine learning models
• AGN with massive black holes have closer galactic neighbors: k-Nearest-Neighbor statistics of an unbiased sample of AGN at z~0.03
• Constraints on the Epoch of Reionization with Roman Space Telescope and the Void Probability Function of Lyman-Alpha Emitters
• Nearest neighbour analysis as a new probe for fuzzy dark matter
• Sahyadri: A simulation suite for the cosmology dependence of the Cosmic Web
• Benchmarking field-level cosmological inference from galaxy redshift surveys
• Skew-spectra: a generalization to spin-$s$
• nuGAN: Generative Adversarial Emulator for Cosmic Web with Neutrinos
• Structure of cosmic web in non-linear regime: the nearest neighbour and spherical contact distributions
• Nearest Neighbour-Based Statistics for 21cm-Galaxy Cross-Correlations in the Epoch of Reionization