The Data Big Bang and the Expanding Digital Universe: High-Dimensional, Complex and Massive Data Sets in an Inflationary Epoch

Abstract

Recent and forthcoming advances in instrumentation, and giant new surveys, are creating astronomical data sets that are not amenable to the methods of analysis familiar to astronomers. Traditional methods are often inadequate not merely because of the size in bytes of the data sets, but also because of the complexity of modern data sets. Mathematical limitations of familiar algorithms and techniques in dealing with such data sets create a critical need for new paradigms for the representation, analysis and scientific visualization (as opposed to illustrative visualization) of heterogeneous, multiresolution data across application domains. Some of the problems presented by the new data sets have been addressed by other disciplines such as applied mathematics, statistics and machine learning and have been utilized by other sciences such as space-based geosciences. Unfortunately, valuable results pertaining to these problems are mostly to be found only in publications outside of astronomy. Here we offer brief overviews of a number of concepts, techniques and developments, some "old" and some new. These are generally unknown to most of the astronomical community, but are vital to the analysis and visualization of complex datasets and images. In order for astronomers to take advantage of the richness and complexity of the new era of data, and to be able to identify, adopt, and apply new solutions, the astronomical community needs a certain degree of awareness and understanding of the new concepts. One of the goals of this paper is to help bridge the gap between applied mathematics, artificial intelligence and computer science on the one side and astronomy on the other.

24 pages, 8 Figures, 1 Table. Accepted for publication: "Advances in Astronomy, special issue "Robotic Astronomy"

References in corpus (13)

• The Palomar Transient Factory: System Overview, Performance and First Results
• Compressed Sensing in Astronomy
• Integrated Sachs-Wolfe effect from the cross correlation of WMAP3 year and the NRAO VLA sky survey data: New results and constraints on dark energy
• The Fast Fourier Transform Telescope
• Point Sources from a Spitzer IRAC Survey of the Galactic Center
• Needlet Detection of Features in WMAP CMB Sky and the Impact on Anisotropies and Hemispherical Asymmetries
• CMB power spectrum estimation using wavelets
• Mining the SDSS archive. I. Photometric redshifts in the nearby universe
• The Palomar-Quest Digital Synoptic Sky Survey
• Exploiting Low-Dimensional Structure in Astronomical Spectra
• PCA of the spectral energy distribution and emission line properties of red 2MASS AGN
• A Cometary Bow Shock and Mid-Infrared Emission Variations Revealed in Spitzer Observations of HD 34078 and IC 405
• Visualization, Exploration and Data Analysis of Complex Astrophysical Data

Cited by in corpus (7)

• The weirdest SDSS galaxies: results from an outlier detection algorithm
• Machine Learning applied to asteroid dynamics
• Foreword to the Focus Issue on Machine Learning in Astronomy and Astrophysics
• A new automated tool for the spectral classification of OB stars
• 3D Detection and Characterisation of ALMA Sources through Deep Learning
• Blueshifted hydrogen emission and shock wave of RR Lyrae variables in SDSS and LAMOST
• A machine learning approach to estimate mid-infrared fluxes from WISE data