Methods for Rapidly Processing Angular Masks of Next-Generation Galaxy Surveys

Abstract

As galaxy surveys become larger and more complex, keeping track of the completeness, magnitude limit, and other survey parameters as a function of direction on the sky becomes an increasingly challenging computational task. For example, typical angular masks of the Sloan Digital Sky Survey contain about N=300,000 distinct spherical polygons. Managing masks with such large numbers of polygons becomes intractably slow, particularly for tasks that run in time O(N^2) with a naive algorithm, such as finding which polygons overlap each other. Here we present a "divide-and-conquer" solution to this challenge: we first split the angular mask into predefined regions called "pixels," such that each polygon is in only one pixel, and then perform further computations, such as checking for overlap, on the polygons within each pixel separately. This reduces O(N^2) tasks to O(N), and also reduces the important task of determining in which polygon(s) a point on the sky lies from O(N) to O(1), resulting in significant computational speedup. Additionally, we present a method to efficiently convert any angular mask to and from the popular HEALPix format. This method can be generically applied to convert to and from any desired spherical pixelization. We have implemented these techniques in a new version of the mangle software package, which is freely available at http://space.mit.edu/home/tegmark/mangle/, along with complete documentation and example applications. These new methods should prove quite useful to the astronomical community, and since mangle is a generic tool for managing angular masks on a sphere, it has the potential to benefit terrestrial mapmaking applications as well.

New version 2.1 of the mangle software now available at http://space.mit.edu/home/tegmark/mangle/ - includes galaxy survey masks and galaxy lists for the latest SDSS data release and the 2dFGRS final data release as well as extensive documentation and examples. 14 pages, 9 figures, matches version accepted by MNRAS

References in corpus (38)

• Maps of Dust IR Emission for Use in Estimation of Reddening and CMBR Foregrounds
• The Sloan Digital Sky Survey: Technical Summary
• Wilkinson Microwave Anisotropy Probe (WMAP) Three Year Results: Implications for Cosmology
• HEALPix -- a Framework for High Resolution Discretization, and Fast Analysis of Data Distributed on the Sphere
• The 2dF Galaxy Redshift Survey: Spectra and redshifts
• The 3D power spectrum of galaxies from the SDSS
• Cosmological Constraints from the SDSS Luminous Red Galaxies
• NYU-VAGC: a galaxy catalog based on new public surveys
• The Fifth Data Release of the Sloan Digital Sky Survey
• MASTER of the CMB Anisotropy Power Spectrum: A Fast Method for Statistical Analysis of Large and Complex CMB Data Sets
• The PSCz Catalogue
• Clustering of High Redshift ($z\ge 2.9$) Quasars from the Sloan Digital Sky Survey
• Systematic errors in weak lensing: application to SDSS galaxy-galaxy weak lensing
• Environmental Dependence of Properties of Galaxies in the Sloan Digital Sky Survey
• The shape of the SDSS DR5 galaxy power spectrum
• Large Synoptic Survey Telescope: Overview
• The power spectrum of galaxies in the 2dF 100k redshift survey
• The Pseudo-$C_l$ method: Cosmic microwave background anisotropy power spectrum statistics for high precision cosmology
• Cosmic microwave background anisotropy power spectrum statistics for high precision cosmology
• A high redshift detection of the integrated Sachs-Wolfe effect
• A Galaxy Photometric Redshift Catalog for the Sloan Digital Sky Survey Data Release 6
• The APM Galaxy Survey III: An Analysis of Systematic Errors in the Angular Correlation Function and Cosmological Implications
• Topology Analysis of the Sloan Digital Sky Survey: I. Scale and Luminosity Dependences
• Correlating the CMB with Luminous Red Galaxies : The Integrated Sachs-Wolfe Effect
• A scheme to deal accurately and efficiently with complex angular masks in galaxy surveys
• SDSS Galaxy Clustering: Luminosity & Colour Dependence and Stochasticity
• MegaZ-LRG: A photometric redshift catalogue of one million SDSS Luminous Red Galaxies
• Cross-correlation of 2MASS and WMAP3: Implications for the Integrated Sachs-Wolfe effect
• Bispectrum and Nonlinear Biasing of Galaxies: Perturbation Analysis, Numerical Simulation and SDSS Galaxy Clustering
• MAPCUMBA : a fast iterative multi-grid map-making algorithm for CMB experiments
• The cross-correlation between galaxies of different luminosities and Colors
• Evidence for an Extended SZ Effect in WMAP Data
• Photometric Redshifts for the Dark Energy Survey and VISTA and Implications for Large Scale Structure
• The DEEP2 Galaxy Redshift Survey: The Evolution of Void Statistics from z~1 to z~0
• SZE Signals in Cluster Models
• Fourier Phase Analysis of SDSS Galaxies
• Searching for modified gravity with baryon oscillations: from SDSS to wide field multiobject spectroscopy (WFMOS)
• CMB Power Spectrum Estimation via Hierarchical Decomposition

Cited by in corpus (158)

• The Baryon Oscillation Spectroscopic Survey of SDSS-III
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Baryon Acoustic Oscillations in the Data Release 10 and 11 galaxy samples
• The Clustering of the SDSS DR7 Main Galaxy Sample I: A 4 per cent Distance Measure at z=0.15
• The Eighth Data Release of the Sloan Digital Sky Survey: First Data from SDSS-III
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Baryon Acoustic Oscillations in the Data Release 9 Spectroscopic Galaxy Sample
• redMaPPer I: Algorithm and SDSS DR8 Catalog
• Eight New Milky Way Companions Discovered in First-Year Dark Energy Survey Data
• PRIMUS: Constraints on Star Formation Quenching and Galaxy Merging, and the Evolution of the Stellar Mass Function From z=0-1
• The Dark Energy Survey Data Release 1
• The Atacama Cosmology Telescope: Sunyaev-Zel'dovich Selected Galaxy Clusters at 148 GHz from Three Seasons of Data
• The clustering of the SDSS-IV extended Baryon Oscillation Spectroscopic Survey DR14 quasar sample: First measurement of Baryon Acoustic Oscillations between redshift 0.8 and 2.2
• The redMaPPer Galaxy Cluster Catalog From DES Science Verification Data
• The PRIsm MUlti-Object Survey (PRIMUS) I: Survey Overview and Characteristics
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: mock galaxy catalogues for the BOSS Final Data Release
• The Dark Energy Survey Data Release 2
• Bayesian physical reconstruction of initial conditions from large scale structure surveys
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: a large sample of mock galaxy catalogues
• Stellar Streams Discovered in the Dark Energy Survey
• The clustering of massive galaxies at z~0.5 from the first semester of BOSS data
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Analysis of potential systematics
• The Clustering of the SDSS Main Galaxy Sample II: Mock galaxy catalogues and a measurement of the growth of structure from Redshift Space Distortions at $z=0.15$
• The Dark Energy Survey Image Processing Pipeline
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Measuring D_A and H at z=0.57 from the Baryon Acoustic Peak in the Data Release 9 Spectroscopic Galaxy Sample
• The clustering of intermediate redshift quasars as measured by the Baryon Oscillation Spectroscopic Survey
• Dark Energy Survey Year 3 Results: Photometric Data Set for Cosmology
• The SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Quasar Target Selection
• The clustering of galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological implications of the large-scale two-point correlation function
• The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Large-scale Structure Catalogs for Cosmological Analysis
• Mapping stellar content to dark matter halos using galaxy clustering and galaxy-galaxy lensing in the SDSS DR7
• Mock galaxy catalogs using the quick particle mesh method
• Milky Way Satellite Census. I. The Observational Selection Function for Milky Way Satellites in DES Y3 and Pan-STARRS DR1
• Eight new luminous z > 6 quasars selected via SED model fitting of VISTA, WISE and Dark Energy Survey Year 1 Observations
• The clustering of galaxies at z~0.5 in the SDSS-III Data Release 9 BOSS-CMASS sample: a test for the LCDM cosmology
• The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: 1000 multi-tracer mock catalogues with redshift evolution and systematics for galaxies and quasars of the final data release
• Clustering of quasars in SDSS-IV eBOSS : study of potential systematics and bias determination
• The miniJPAS survey: a preview of the Universe in 56 colours
• The DECam Local Volume Exploration Survey: Overview and First Data Release
• Clues on void evolution II: Measuring density and velocity profiles on SDSS galaxy redshift space distortions
• Shadows in the Dark: Low-Surface-Brightness Galaxies Discovered in the Dark Energy Survey
• SDSS Galaxy Clustering: Luminosity & Colour Dependence and Stochasticity
• Constraints on cosmic hemispherical power anomalies from quasars
• Heavily Reddened Quasars at z~2 in the UKIDSS Large Area Survey: A Transitional Phase in AGN Evolution
• No galaxy left behind: accurate measurements with the faintest objects in the Dark Energy Survey
• Galaxy clustering, photometric redshifts and diagnosis of systematics in the DES Science Verification data
• A robust public catalogue of voids and superclusters in the SDSS Data Release 7 galaxy surveys
• Estimating the large-scale angular power spectrum in the presence of systematics: a case study of Sloan Digital Sky Survey quasars
• Cross-correlations as a cosmological carbon monoxide detector
• Cosmic Voids in the SDSS DR12 BOSS Galaxy Sample: The Alcock-Paczynski Test
• DES J0454-4448: Discovery of the First Luminous z > 6 Quasar from the Dark Energy Survey
• The Clustering of MgII Absorption Systems at z=0.5 and Detection of Cold Gas in Massive Halos
• A Cosmic Void Catalog of SDSS DR12 BOSS Galaxies
• The Stripe 82 Massive Galaxy Project II: Stellar Mass Completeness of Spectroscopic Galaxy Samples from the Baryon Oscillation Spectroscopic Survey
• The clustering of luminous red galaxies at z $\sim$ 0.7 from eBOSS and BOSS data
• The Angular Clustering of Infrared-Selected Obscured and Unobscured Quasars
• Alignments of Galaxy Group Shapes with Large Scale Structure
• A 14 $h^{-3}$ Gpc${}^3$ study of cosmic homogeneity using BOSS DR12 quasar sample
• PRIMUS: Galaxy Clustering as a Function of Luminosity and Color at 0.2<z<1
• Mapping and simulating systematics due to spatially-varying observing conditions in DES Science Verification data
• The Sloan Digital Sky Survey Peculiar Velocity Catalogue
• Cosmological Measurements from Angular Power Spectra Analysis of BOSS DR12 Tomography
• SPIDERS: Selection of spectroscopic targets using AGN candidates detected in all-sky X-ray surveys
• Efficient Reconstruction of Linear Baryon Acoustic Oscillations in Galaxy Surveys
• The Redshift-Space Cluster-Galaxy Cross-Correlation Function: I. Modeling Galaxy Infall onto Millennium Simulation Clusters and SDSS Groups
• Exploiting the full potential of photometric quasar surveys: Optimal power spectra through blind mitigation of systematics
• The Clustering of High-Redshift (2.9 $\le$ z $\le$ 5.1) Quasars in SDSS Stripe 82
• The Phoenix stream: a cold stream in the Southern hemisphere
• Methods for Bayesian power spectrum inference with galaxy surveys
• Systematic effects in large-scale angular power spectra of photometric quasars and implications for constraining primordial nongaussianity
• Detecting the Cosmic Dipole Anisotropy in Large-Scale Radio Surveys
• Direct Minkowski Functional analysis of large redshift surveys: a new high--speed code tested on the luminous red galaxy Sloan Digital Sky Survey-DR7 catalogue
• The LMC geometry and outer stellar populations from early DES data
• An improved measurement of baryon acoustic oscillations from the correlation function of galaxy clusters at $z\sim 0.3$
• The Stripe 82 Massive Galaxy Project I: Catalog Construction
• COSMIC BIRTH: Efficient Bayesian Inference of the Evolving Cosmic Web from Galaxy Surveys
• New constraints on ${\sigma }_8$ from a joint analysis of stacked gravitational lensing and clustering of galaxy clusters
• Large Covariance Matrices: Smooth Models from the 2-Point Correlation Function
• The characteristic halo masses of half-a-million WISE-selected quasars
• Properties of satellite galaxies in the SDSS photometric survey: luminosities, colours and projected number density profiles
• Galaxy Counts on the CMB Cold Spot
• Halo Histories vs. Galaxy Properties at z=0, I: The Quenching of Star Formation
• Updated measurements of the dark matter halo masses of obscured quasars with improved WISE and Planck data
• Updated Results on the Galaxy-Halo Connection from Satellite Kinematics in SDSS
• The galaxy counts-in-cells distribution from the SDSS
• Integrated approach to cosmology: Combining CMB, large-scale structure and weak lensing
• FLIMFLAM DR1: The First Constraints on the Cosmic Baryon Distribution from 8 FRB sightlines
• Measuring the distance-redshift relation with the baryon acoustic oscillations of galaxy clusters
• J-PLUS: Unveiling the brightest-end of the Lyα luminosity function at 2.0<z<3.3 over 1000 deg^2
• Weak lensing magnification in the Dark Energy Survey Science Verification Data
• $\mathrm{S}\mathrm{IM}\mathrm{BIG}$: Mock Challenge for a Forward Modeling Approach to Galaxy Clustering
• The best fit for the observed galaxy Counts-in-Cell distribution function
• Matrix-free Large Scale Bayesian inference in cosmology
• Brightest galaxies as halo centre tracers in SDSS DR7
• The ALHAMBRA survey: evolution of galaxy clustering since $z\sim 1$
• The dependence of galaxy clustering on tidal environment in the Sloan Digital Sky Survey
• Background sky obscuration by cluster galaxies as a source of systematic error for weak lensing
• The SDSS-IV eBOSS: emission line galaxy catalogues at z=0.8 and study of systematic errors in the angular clustering
• Halo Occupation Distribution Modeling of Green Valley Galaxies
• Search for primordial non-Gaussianity in the quasars of SDSS-III BOSS DR9
• Correcting correlation functions for redshift-dependent interloper contamination
• Galaxy and Mass Assembly (GAMA): Colour and luminosity dependent clustering from calibrated photometric redshifts
• Cosmological Density Fluctuations on 100Mpc Scales and their ISW Effect
• The UV-Optical Color Dependence of Galaxy Clustering in the Local Universe
• C${}^3$-Cluster Clustering Cosmology I. New constraints on the cosmic growth rate at z~0.3 from redshift-space clustering anisotropies
• Evidence of Galaxy Assembly Bias in SDSS DR7 Galaxy Samples from Count Statistics
• The Integrated Sachs-Wolfe Signal from BOSS Super-Structures
• The Clustering of Galaxies in the Completed SDSS-III Baryon Oscillation Spectroscopic Survey: Cosmic Flows and Cosmic Web from Luminous Red Galaxies
• Measurement of $H(z)$ and $D_A(z)$ from the two-dimensional power spectrum of Sloan Digital Sky Survey luminous red galaxies
• STRIDES: Spectroscopic and photometric characterization of the environment and effects of mass along the line of sight to the gravitational lenses DES J0408-5354 and WGD 2038-4008
• The co-evolution of black hole growth and star formation from a cross-correlation analysis between quasars and the cosmic infrared background
• Modelling the Milky Way. I -- Method and first results fitting the thick disk and halo with DES-Y3 data
• Host Dark Matter Halos of WISE-selected Obscured & Unobscured Quasars: Evidence for Evolution
• C${}^3$-Cluster Clustering Cosmology II. First detection of the BAO peak in the three-point correlation function of galaxy clusters
• A Deep Proper Motion Catalog Within The Sloan Digital Sky Survey Footprint
• Comparing Dark Energy Survey and HST-CLASH observations of the galaxy cluster RXC J2248.7-4431: implications for stellar mass versus dark matter
• Galaxy and Mass Assembly (GAMA): Small-scale anisotropic galaxy clustering and the pairwise velocity dispersion of galaxies
• Maturing Satellite Kinematics into a Competitive Probe of the Galaxy-Halo Connection
• Measurement of Redshift Space Power Spectrum for BOSS galaxies and the Growth Rate at redshift 0.57
• A finer view of the conditional galaxy luminosity function and magnitude-gap statistics
• Halo Histories vs. Galaxy Properties at z=0, III: The Properties of Star-Forming Galaxies
• Bayesian power-spectrum inference with foreground and target contamination treatment
• The power spectrum from the angular distribution of galaxies in the CFHTLS-Wide fields at redshift ~0.7
• Full forward model of galaxy clustering statistics with AbacusSummit lightcones
• Mass Functions, Luminosity Functions, and Completeness Measurements from Clustering Redshifts
• The Global Environment of Small Galaxy Systems
• A Halo Occupation Interpretation Of Quasars At $z\sim 1.5$ Using Very Small Scale Clustering Information
• CMASS galaxy sample and the ontological status of the cosmological principle
• Tomographic Constraints on Gravity from Angular Redshift Fluctuations in the Late Universe
• Group galaxy number density profiles far out: is the 'one-halo' term NFW out to $>10$ virial radii?
• Host Dark Matter Halos of SDSS Red and Blue Quasars: No Significant Difference in Large-scale Environment
• Large-scale retrospective relative spectro-photometric self-calibration in space
• Compatibility of the Large Quasar Groups with the Concordance Cosmological Model
• Wide Area VISTA Extra-galactic Survey (WAVES): Unsupervised star-galaxy separation on the WAVES-Wide photometric input catalogue using UMAP and $\mathrm{HDBSCAN}$
• The ALHAMBRA survey: evolution of galaxy spectral segregation
• On the nature of Small Galaxy Systems
• Halo Masses of MgII absorbers at z\sim 0.5 from SDSS DR7
• The XMM Cluster Survey: The Halo Occupation Number of BOSS galaxies in X-ray clusters
• The Impact of the Dusty Torus on Obscured Quasar Halo Mass Measurements
• Galaxy and Mass Assembly (GAMA): the clustering of galaxy groups
• PAC.V. The Roles of Mass and Environment in the Quenching of Galaxies
• PRIMUS: The Effect of Physical Scale on the Luminosity-Dependence of Galaxy Clustering via Cross-Correlations
• The properties and environment of very young galaxies in the local Universe
• Constraining cosmological parameters using void statistics from the SDSS survey
• Euclid preparation: VI. Verifying the Performance of Cosmic Shear Experiments
• Systematic Survey of the Correlation between Northern HECR Events and SDSS Galaxies
• A $K_s$-band selected catalogue of objects in the ALHAMBRA survey
• Stellar Population Properties of ETGs in Compact Groups of Galaxies
• Joint constraints on galaxy bias and ${\sigma }_8$ through the N-pdf of the galaxy number density
• Inferring Cosmological Parameters on SDSS via Domain-Generalized Neural Networks and Lightcone Simulations
• Efficient data structures for masks on 2D grids
• Cosmic Census: Relative Distributions of Dark Matter, Galaxies and Diffuse Gas
• Dark Energy Survey Year 1 Results: Photometric Data Set for Cosmology
• Counting quasar--radio source pairs to derive the millijansky radio luminosity function and clustering strength to z=3.5
• ZTF SN Ia DR2: Exploring SN Ia properties in the vicinity of under-dense environments
• Dark Energy Survey Year 3 Results: Galaxy Sample for BAO Measurement
• Covariances of Galaxy Stellar Mass Functions and Correlation Functions
• The Kaiser-Rocket effect: three decades and counting
• Skykatana: a scalable framework to construct sky masks for the Vera Rubin Observatory and large astronomical surveys
• Measuring peculiar velocity and tomographic redshift dipole with DESI DR1 catalogs